Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.....................X..............X........X..........] duration: 7.994792293s FAIL (2025-08-07 23:55:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 408ba8db7a0d8af05b7d5f8336d5c5f7f4f0063d72c98ffee3c421a5da299777 != before d0b3d19e255585f56d510f04d387098fcb83db6017871f75c2e11245b6143455 ======LOGS====== ished. err: <nil> Aug 7 23:48:24.457116068 (14h8m24.046461055s) [upload=14655]: Starting upload Aug 7 23:48:24.457179188 (14h8m24.046524175s) [upload=14655]: Done starting upload Aug 7 23:48:24.457293146 (14h8m24.046638133s) [upload=14655]: Uploading segments... Aug 7 23:48:24.457654307 (14h8m24.046999294s) [upload=14655]: Got next segment. Inline: false Aug 7 23:48:24.457709964 (14h8m24.047054951s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:24.457715013 (14h8m24.04706s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:48:24.901153105 (14h8m24.490498082s) [seg_pos={0 0}, upload=14655]: Begin upload segment... Aug 7 23:48:24.901189916 (14h8m24.490534903s) [seg_pos={0 0}, upload=14655]: Done begin upload segment. Aug 7 23:48:24.901208150 (14h8m24.490553137s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901547320 (14h8m24.490892297s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901866269 (14h8m24.491211257s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901979486 (14h8m24.491324473s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902167115 (14h8m24.491512102s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902393378 (14h8m24.491738355s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902443794 (14h8m24.491788771s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902663996 (14h8m24.492008983s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902931768 (14h8m24.492276755s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903278531 (14h8m24.492623518s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903535794 (14h8m24.492880781s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903745445 (14h8m24.493090432s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903975075 (14h8m24.493320062s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904281360 (14h8m24.493626337s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904481643 (14h8m24.49382663s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904668631 (14h8m24.494013608s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904852633 (14h8m24.49419762s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.905040943 (14h8m24.49438593s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:26.003752552 (14h8m25.593097539s) [upload=14655]: Next returned nil segment Aug 7 23:48:26.003765798 (14h8m25.593110785s) [upload=14655]: Waiting for error group managing segments... Aug 7 23:48:26.282577597 (14h8m25.871922574s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.322820552 (14h8m25.912165529s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.366928523 (14h8m25.95627351s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.822974095 (14h8m26.412319082s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.895356812 (14h8m26.4847018s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.915282265 (14h8m26.504627252s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.929180326 (14h8m26.518525313s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.126611753 (14h8m26.71595674s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.271660348 (14h8m26.861005335s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.274412033 (14h8m26.86375702s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.336714535 (14h8m26.926059522s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.759468572 (14h8m27.348813559s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.904353465 (14h8m27.493698452s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960895364 (14h8m27.550240341s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960922416 (14h8m27.550267403s) [seg_pos={0 0}, upload=14655]: Segment reached optimal threshold of 14 pieces. Aug 7 23:48:27.961026686 (14h8m27.550371673s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961266504 (14h8m27.550611491s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961284479 (14h8m27.550629456s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: rpc: dial tcp 104.36.82.5:4869: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:48:27.961415680 (14h8m27.550760667s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC): protocol: context canceled; context canceled Aug 7 23:48:27.961429706 (14h8m27.550774693s) [seg_pos={0 0}, upload=14655]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:48:27.961507495 (14h8m27.550852482s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:48:27.961511613 (14h8m27.5508566s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:28.311226257 (14h8m27.900571244s) [upload=14655]: Done uploading segments... Aug 7 23:48:28.311246106 (14h8m27.900591083s) [upload=14655]: Upload finished. err: <nil> Duration: 7.994816982s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............XX.X.........................................X.X] duration: 1m0.001154337s FAIL (2025-08-07 23:55:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== UVDXIUWWWZMM3Q, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:04.510384749 (14h14m4.099729736s) [noise=true , address=104.36.82.41:4776 , piece_id=QHU2MMJANR77RUDKDGJG4CBANZJ5EES36V3LPFR44L2NOADPZW7A, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:04.510678430 (14h14m4.100023417s) [noise=true , address=104.36.82.62:4412 , piece_id=JB7TZM3QJTAOZ3LOQDTHORO5RX5FI4CHIHDRKBZWHXM6726L6KZQ, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:04.510696575 (14h14m4.100041552s) [noise=true , address=45.91.116.5:4313 , piece_id=DMNO7W6TC25LWVYNLCWL4C6R6B6NJAXL3BJC7LMO5GYDWJXHWZOA, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:04.510933339 (14h14m4.100278326s) [noise=true , address=104.36.82.43:4840 , piece_id=GFMDT247SEVYVNGCTKO75KEQQI2VL33S2XSJGBOQCZFTU7TJT4HQ, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:04.511183387 (14h14m4.100528374s) [noise=true , address=104.36.82.35:4608 , piece_id=PFB4PYEONSY3U2SYSJOC3KU4SGHF5XF5Q2JRWOKYBFDLKFJT344Q, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:04.511423776 (14h14m4.100768763s) [noise=true , address=64.49.13.140:4514 , piece_id=N5WKZPIZIM4P3GLXBEQSAQDBPSPZ3NUD7OH2XILUTQGENOFJ46UQ, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:04.511695226 (14h14m4.101040213s) [noise=true , address=79.127.219.51:4181 , piece_id=LJ7T5J2SKZMOJP5NH4J7MALDEOC7HCU4O4N4EFMRKXXCJHCWD6WA, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:04.512233786 (14h14m4.101578763s) [noise=true , address=104.36.82.16:4299 , piece_id=VTGF3XXVVHN26NXP3NJTO7VEQICK7XONWMMQKQOMIUB4VEZELK6Q, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:04.512490708 (14h14m4.101835695s) [noise=true , address=79.127.205.233:4625 , piece_id=QDT34LKAAKP5JAQE4PIMBEJIOLEAKVX5MZLDCXMXFL22XCYFCLXQ, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:04.512742199 (14h14m4.102087176s) [noise=true , address=104.36.82.37:4686 , piece_id=3BNAZZCWHME7WTX6WNE6WALK2LVCAAMOXX5DWT5W7T4GXKMYMHWQ, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:04.512996827 (14h14m4.102341814s) [noise=true , address=104.36.82.58:4291 , piece_id=S7EJWGTHRSWTUKVK7G4RNV67DAZQEPLSLGT7PXKQKPT7QRWRLGAA, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:04.513214392 (14h14m4.102559369s) [noise=true , address=45.91.116.8:4074 , piece_id=DKNSQTDYOM52SGWDXWUECY4FDWB5SLWCIEBJSHWJ6RI5NFXKZLLQ, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:04.514005997 (14h14m4.103350994s) [noise=true , address=104.36.82.5:4861 , piece_id=7XPGF2JQGAESQ7C4VT23B6F2VTZJV2AM5QSTTP6E7TGUOYF5ECPQ, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:04.514283968 (14h14m4.103628955s) [noise=true , address=64.49.13.154:4936 , piece_id=PB4PRHFNGVK2M6ADPDXUUTNAWUFE5FSG4VHMNJPLXHNSMDXL77AQ, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:04.514537404 (14h14m4.103882381s) [noise=true , address=104.36.82.59:4330 , piece_id=3NMOBCIZJ3H4KXETZB3QMBWKKX6MTWBW4AQ65AXGQ7EQMFUHFMXQ, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:04.514816889 (14h14m4.104161876s) [noise=true , address=104.36.82.4:4823 , piece_id=NET3MIJVOHYUALAIYJOIKVDHOLOPPA2GQTRAXBVT6M5HTTQZXDVQ, seg_pos={0 0}, upload=14752]: Uploading piece... Aug 7 23:54:05.635647276 (14h14m5.224992263s) [upload=14752]: Next returned nil segment Aug 7 23:54:05.635666933 (14h14m5.22501193s) [upload=14752]: Waiting for error group managing segments... Aug 7 23:54:05.922526748 (14h14m5.511871725s) [noise=true , address=64.49.13.154:4936 , piece_id=PB4PRHFNGVK2M6ADPDXUUTNAWUFE5FSG4VHMNJPLXHNSMDXL77AQ, seg_pos={0 0}, upload=14752]: Done uploading piece. err: <nil> Aug 7 23:54:05.947370682 (14h14m5.536715669s) [noise=true , address=79.127.205.233:4625 , piece_id=QDT34LKAAKP5JAQE4PIMBEJIOLEAKVX5MZLDCXMXFL22XCYFCLXQ, seg_pos={0 0}, upload=14752]: Done uploading piece. err: <nil> Aug 7 23:54:05.980419323 (14h14m5.5697643s) [noise=true , address=64.49.13.140:4514 , piece_id=N5WKZPIZIM4P3GLXBEQSAQDBPSPZ3NUD7OH2XILUTQGENOFJ46UQ, seg_pos={0 0}, upload=14752]: Done uploading piece. err: <nil> Aug 7 23:54:06.414054772 (14h14m6.003399759s) [noise=true , address=79.127.201.215:4347 , piece_id=SNMRNPV4JJBYKECO4W2H6V233REK65XKLFPCH5UVDXIUWWWZMM3Q, seg_pos={0 0}, upload=14752]: Done uploading piece. err: <nil> Aug 7 23:54:06.450707258 (14h14m6.040052265s) [noise=true , address=104.36.82.35:4608 , piece_id=PFB4PYEONSY3U2SYSJOC3KU4SGHF5XF5Q2JRWOKYBFDLKFJT344Q, seg_pos={0 0}, upload=14752]: Done uploading piece. err: <nil> Aug 7 23:54:06.451811450 (14h14m6.041156448s) [noise=true , address=79.127.219.51:4181 , piece_id=LJ7T5J2SKZMOJP5NH4J7MALDEOC7HCU4O4N4EFMRKXXCJHCWD6WA, seg_pos={0 0}, upload=14752]: Done uploading piece. err: <nil> Aug 7 23:54:06.549863342 (14h14m6.139208329s) [noise=true , address=104.36.82.16:4299 , piece_id=VTGF3XXVVHN26NXP3NJTO7VEQICK7XONWMMQKQOMIUB4VEZELK6Q, seg_pos={0 0}, upload=14752]: Done uploading piece. err: <nil> Aug 7 23:54:06.577009030 (14h14m6.166354017s) [noise=true , address=45.91.116.5:4313 , piece_id=DMNO7W6TC25LWVYNLCWL4C6R6B6NJAXL3BJC7LMO5GYDWJXHWZOA, seg_pos={0 0}, upload=14752]: Done uploading piece. err: <nil> Aug 7 23:54:06.604883862 (14h14m6.194228849s) [noise=true , address=104.36.82.43:4840 , piece_id=GFMDT247SEVYVNGCTKO75KEQQI2VL33S2XSJGBOQCZFTU7TJT4HQ, seg_pos={0 0}, upload=14752]: Done uploading piece. err: <nil> Aug 7 23:54:06.617381383 (14h14m6.20672636s) [noise=true , address=104.36.82.41:4776 , piece_id=QHU2MMJANR77RUDKDGJG4CBANZJ5EES36V3LPFR44L2NOADPZW7A, seg_pos={0 0}, upload=14752]: Done uploading piece. err: <nil> Aug 7 23:54:06.673028762 (14h14m6.262373739s) [noise=true , address=104.36.82.61:4406 , piece_id=6YAAOBIT2VSE7ZDMGJPF4EF5FZNLXT5XZXRL7DPOGELOIXSDIACA, seg_pos={0 0}, upload=14752]: Done uploading piece. err: <nil> Aug 7 23:54:06.675718599 (14h14m6.265063586s) [noise=true , address=45.91.116.8:4074 , piece_id=DKNSQTDYOM52SGWDXWUECY4FDWB5SLWCIEBJSHWJ6RI5NFXKZLLQ, seg_pos={0 0}, upload=14752]: Done uploading piece. err: <nil> Aug 7 23:54:06.722640694 (14h14m6.311985671s) [noise=true , address=104.36.82.59:4330 , piece_id=3NMOBCIZJ3H4KXETZB3QMBWKKX6MTWBW4AQ65AXGQ7EQMFUHFMXQ, seg_pos={0 0}, upload=14752]: Done uploading piece. err: <nil> Aug 7 23:54:06.767981284 (14h14m6.357326271s) [noise=true , address=104.36.82.4:4823 , piece_id=NET3MIJVOHYUALAIYJOIKVDHOLOPPA2GQTRAXBVT6M5HTTQZXDVQ, seg_pos={0 0}, upload=14752]: Done uploading piece. err: <nil> Aug 7 23:54:06.768007514 (14h14m6.357352501s) [seg_pos={0 0}, upload=14752]: Segment reached optimal threshold of 14 pieces. Aug 7 23:54:06.768130910 (14h14m6.357475887s) [noise=true , address=104.36.82.5:4861 , piece_id=7XPGF2JQGAESQ7C4VT23B6F2VTZJV2AM5QSTTP6E7TGUOYF5ECPQ, seg_pos={0 0}, upload=14752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS): protocol: expected piece hash; context canceled; EOF Aug 7 23:54:06.768241752 (14h14m6.357586739s) [noise=true , address=104.36.82.58:4291 , piece_id=S7EJWGTHRSWTUKVK7G4RNV67DAZQEPLSLGT7PXKQKPT7QRWRLGAA, seg_pos={0 0}, upload=14752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gn31FU1BnbHPLvB7k1RZhtJbDdJCLyBLa5A6VhDNMFN7erSG1): protocol: context canceled; context canceled Aug 7 23:54:06.768410756 (14h14m6.357755743s) [noise=true , address=104.36.82.62:4412 , piece_id=JB7TZM3QJTAOZ3LOQDTHORO5RX5FI4CHIHDRKBZWHXM6726L6KZQ, seg_pos={0 0}, upload=14752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vBvEfCYZxsv17cVsQTbCQ4YjZfiD4RXjWyz7Ka9ZZrtvzC7SS): protocol: write tcp 188.166.225.170:44966->104.36.82.62:4412: use of closed network connection; write tcp 188.166.225.170:44966->104.36.82.62:4412: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44966->104.36.82.62:4412: use of closed network connection Aug 7 23:54:06.768427698 (14h14m6.357772686s) [noise=true , address=104.36.82.37:4686 , piece_id=3BNAZZCWHME7WTX6WNE6WALK2LVCAAMOXX5DWT5W7T4GXKMYMHWQ, seg_pos={0 0}, upload=14752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U61X2oPRJjzWTuGEXqjModoHVsXvmjsPbftqGAfE5ff62Yzpp): protocol: rpc: dial tcp 104.36.82.37:4686: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:54:06.768447726 (14h14m6.357792713s) [seg_pos={0 0}, upload=14752]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:54:06.768545904 (14h14m6.357890881s) [upload=14752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:54:06.768550573 (14h14m6.35789555s) [upload=14752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:07.048676413 (14h14m6.6380214s) [upload=14752]: Done uploading segments... Aug 7 23:54:07.048723864 (14h14m6.638068851s) [upload=14752]: Upload finished. err: <nil> Aug 7 23:54:27.086586892 (14h14m26.675931869s) [upload=14759]: Starting upload Aug 7 23:54:27.086643400 (14h14m26.675988388s) [upload=14759]: Done starting upload Aug 7 23:54:27.086718765 (14h14m26.676063752s) [upload=14759]: Uploading segments... Aug 7 23:54:27.086838093 (14h14m26.67618308s) [upload=14759]: Got next segment. Inline: false Aug 7 23:54:27.086868591 (14h14m26.676213568s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:27.086872900 (14h14m26.676217887s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:55:26.443844292 (14h15m26.033189269s) [upload=14759]: Next segment err: context deadline exceeded Aug 7 23:55:26.443911349 (14h15m26.033256326s) [upload=14759]: Done uploading segments... Aug 7 23:55:26.443945374 (14h15m26.033290351s) [upload=14759]: Upload finished. err: context deadline exceeded Duration: 1m0.001174365s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [..........X.....X.X...................................X.....] duration: 2.313355168s FAIL (2025-08-07 23:55:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.313379254s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X..X.XXXX.X............................................XX..X] duration: 30.001802339s FAIL (2025-08-07 23:55:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Aug 7 23:54:15.608218293 (14h14m15.19756328s) [seg_pos={0 0}, upload=14754]: Begin upload segment... Aug 7 23:54:15.608257779 (14h14m15.197602766s) [seg_pos={0 0}, upload=14754]: Done begin upload segment. Aug 7 23:54:15.608280372 (14h14m15.197625359s) [noise=true , address=45.91.116.11:4118 , piece_id=7K5ZSIRJDJ7WUUA5QYROV3ECMZ375IJOJVXYESDBTYVZ6U2ESFKA, seg_pos={0 0}, upload=14754]: Uploading piece... Aug 7 23:54:15.608379621 (14h14m15.197724608s) [noise=true , address=77.90.31.15:4698 , piece_id=OGQE7SIQFFOQXBFGS4U23HRNO4MF4KPVW6233EW4COPVDENABAMA, seg_pos={0 0}, upload=14754]: Uploading piece... Aug 7 23:54:15.608629389 (14h14m15.197974376s) [noise=true , address=77.90.31.21:4877 , piece_id=O5M5H46FFPPZVGJFH22ZYMV3MQQGOVNZMROQNRSJQSFGNTX35USQ, seg_pos={0 0}, upload=14754]: Uploading piece... Aug 7 23:54:15.608759809 (14h14m15.198104796s) [noise=true , address=104.36.82.12:4149 , piece_id=CLMPWWI6PYPQ6GKI2GKU2KYGRTGWH4A7RWD2BFG2EN2LQD4Z2XNA, seg_pos={0 0}, upload=14754]: Uploading piece... Aug 7 23:54:15.608941737 (14h14m15.198286724s) [noise=true , address=104.36.82.8:4970 , piece_id=STP277ZYIPFPXZYVJPLAWXTGFBEE5UFMPVB3UADDICHPONXHVBKQ, seg_pos={0 0}, upload=14754]: Uploading piece... Aug 7 23:54:15.609167780 (14h14m15.198512757s) [noise=true , address=77.90.31.19:4843 , piece_id=LWSGZ2HIV7Z6ZSXSSNUJGKLYNZ4H6XHPMM273WBZX6U5I4JXLIAQ, seg_pos={0 0}, upload=14754]: Uploading piece... Aug 7 23:54:15.609200402 (14h14m15.198545389s) [noise=true , address=45.91.116.4:4937 , piece_id=HKJZUQE7XIMBFJAE66VFBQPHVEZLLHQZKKTZGSCTTUTTX7CWNSTA, seg_pos={0 0}, upload=14754]: Uploading piece... Aug 7 23:54:15.609445190 (14h14m15.198790177s) [noise=true , address=104.36.82.9:4005 , piece_id=ZPYATJO7LDYKB2DA47Y27AZLBA6WX6WQX3NN6OXJHGHKDJOOOEAA, seg_pos={0 0}, upload=14754]: Uploading piece... Aug 7 23:54:15.609473775 (14h14m15.198818762s) [noise=true , address=77.90.31.13:4626 , piece_id=GCXI74B7LOXXQEH7ELRG223CRSZKEAWLRCQINAM6XITMU7HNJZQQ, seg_pos={0 0}, upload=14754]: Uploading piece... Aug 7 23:54:15.609687835 (14h14m15.199032812s) [noise=true , address=104.36.82.12:4151 , piece_id=WME5KYWG2SHXNCORNMIXQGVSR56KCMIQ4VLSVC5TZFAFST7VPBXA, seg_pos={0 0}, upload=14754]: Uploading piece... Aug 7 23:54:15.609774791 (14h14m15.199119778s) [noise=true , address=104.36.82.13:4186 , piece_id=EBGNOK6YRD5VVJLN2IRIB35SH6WOYTOFZZKZ67VAMTMAJ6LC52QQ, seg_pos={0 0}, upload=14754]: Uploading piece... Aug 7 23:54:15.609979091 (14h14m15.199324078s) [noise=true , address=77.90.31.14:4662 , piece_id=NQK6BAAKA5ODNVDGMYADY2MOG2JA7KH6SITP5GFH46OK2V4GDWTQ, seg_pos={0 0}, upload=14754]: Uploading piece... Aug 7 23:54:15.610052642 (14h14m15.19939763s) [noise=true , address=104.36.82.11:4114 , piece_id=6LAAGIRGLQUOCMYH4EANUREFTJJ36NLK6RJQUEPVV232WFURCB3A, seg_pos={0 0}, upload=14754]: Uploading piece... Aug 7 23:54:15.610192550 (14h14m15.199537527s) [noise=true , address=104.36.82.5:4839 , piece_id=BZV6ZWMBPWIHLDHHRKN5YDR2GAVMPLGNFYOEE4THHQJDFUEMO5RA, seg_pos={0 0}, upload=14754]: Uploading piece... Aug 7 23:54:15.610263877 (14h14m15.199608864s) [noise=true , address=104.36.82.13:4187 , piece_id=J62C7GV4G3GIWWRZ3ATN5A2HJJ5RMZFYQRQLVQN3J2IGWIFKC6JA, seg_pos={0 0}, upload=14754]: Uploading piece... Aug 7 23:54:15.610379919 (14h14m15.199724906s) [noise=true , address=104.36.82.16:4294 , piece_id=7OFRI3GR7NL3JNK3VFVCGE3TGZGLFCKAN6XX2JYF4VEX3SQATROQ, seg_pos={0 0}, upload=14754]: Uploading piece... Aug 7 23:54:17.369399477 (14h14m16.958744464s) [upload=14754]: Next returned nil segment Aug 7 23:54:17.369417320 (14h14m16.958762307s) [upload=14754]: Waiting for error group managing segments... Aug 7 23:54:17.753316909 (14h14m17.342661896s) [noise=true , address=104.36.82.9:4005 , piece_id=ZPYATJO7LDYKB2DA47Y27AZLBA6WX6WQX3NN6OXJHGHKDJOOOEAA, seg_pos={0 0}, upload=14754]: Done uploading piece. err: <nil> Aug 7 23:54:17.820184256 (14h14m17.409529243s) [noise=true , address=45.91.116.11:4118 , piece_id=7K5ZSIRJDJ7WUUA5QYROV3ECMZ375IJOJVXYESDBTYVZ6U2ESFKA, seg_pos={0 0}, upload=14754]: Done uploading piece. err: <nil> Aug 7 23:54:17.851936326 (14h14m17.441281313s) [noise=true , address=77.90.31.13:4626 , piece_id=GCXI74B7LOXXQEH7ELRG223CRSZKEAWLRCQINAM6XITMU7HNJZQQ, seg_pos={0 0}, upload=14754]: Done uploading piece. err: <nil> Aug 7 23:54:17.860904455 (14h14m17.450249442s) [noise=true , address=104.36.82.11:4114 , piece_id=6LAAGIRGLQUOCMYH4EANUREFTJJ36NLK6RJQUEPVV232WFURCB3A, seg_pos={0 0}, upload=14754]: Done uploading piece. err: <nil> Aug 7 23:54:17.898716290 (14h14m17.488061277s) [noise=true , address=104.36.82.16:4294 , piece_id=7OFRI3GR7NL3JNK3VFVCGE3TGZGLFCKAN6XX2JYF4VEX3SQATROQ, seg_pos={0 0}, upload=14754]: Done uploading piece. err: <nil> Aug 7 23:54:17.898936491 (14h14m17.488281478s) [noise=true , address=104.36.82.5:4839 , piece_id=BZV6ZWMBPWIHLDHHRKN5YDR2GAVMPLGNFYOEE4THHQJDFUEMO5RA, seg_pos={0 0}, upload=14754]: Done uploading piece. err: <nil> Aug 7 23:54:17.919411826 (14h14m17.508756833s) [noise=true , address=104.36.82.8:4970 , piece_id=STP277ZYIPFPXZYVJPLAWXTGFBEE5UFMPVB3UADDICHPONXHVBKQ, seg_pos={0 0}, upload=14754]: Done uploading piece. err: <nil> Aug 7 23:54:17.974977118 (14h14m17.564322105s) [noise=true , address=77.90.31.19:4843 , piece_id=LWSGZ2HIV7Z6ZSXSSNUJGKLYNZ4H6XHPMM273WBZX6U5I4JXLIAQ, seg_pos={0 0}, upload=14754]: Done uploading piece. err: <nil> Aug 7 23:54:18.094194155 (14h14m17.683539132s) [noise=true , address=77.90.31.14:4662 , piece_id=NQK6BAAKA5ODNVDGMYADY2MOG2JA7KH6SITP5GFH46OK2V4GDWTQ, seg_pos={0 0}, upload=14754]: Done uploading piece. err: <nil> Aug 7 23:54:18.099340441 (14h14m17.688685428s) [noise=true , address=45.91.116.4:4937 , piece_id=HKJZUQE7XIMBFJAE66VFBQPHVEZLLHQZKKTZGSCTTUTTX7CWNSTA, seg_pos={0 0}, upload=14754]: Done uploading piece. err: <nil> Aug 7 23:54:18.142675053 (14h14m17.73202005s) [noise=true , address=77.90.31.15:4698 , piece_id=OGQE7SIQFFOQXBFGS4U23HRNO4MF4KPVW6233EW4COPVDENABAMA, seg_pos={0 0}, upload=14754]: Done uploading piece. err: <nil> Aug 7 23:54:18.253606205 (14h14m17.842951192s) [noise=true , address=77.90.31.21:4877 , piece_id=O5M5H46FFPPZVGJFH22ZYMV3MQQGOVNZMROQNRSJQSFGNTX35USQ, seg_pos={0 0}, upload=14754]: Done uploading piece. err: <nil> Aug 7 23:54:20.126049483 (14h14m19.71539447s) [noise=true , address=104.36.82.13:4186 , piece_id=EBGNOK6YRD5VVJLN2IRIB35SH6WOYTOFZZKZ67VAMTMAJ6LC52QQ, seg_pos={0 0}, upload=14754]: Done uploading piece. err: <nil> Aug 7 23:54:20.126082425 (14h14m19.715427412s) [seg_pos={0 0}, upload=14754]: Segment reached optimal threshold of 13 pieces. Aug 7 23:54:20.126309259 (14h14m19.715654246s) [noise=true , address=104.36.82.13:4187 , piece_id=J62C7GV4G3GIWWRZ3ATN5A2HJJ5RMZFYQRQLVQN3J2IGWIFKC6JA, seg_pos={0 0}, upload=14754]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:53406->104.36.82.13:4187: use of closed network connection; write tcp 188.166.225.170:53406->104.36.82.13:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53406->104.36.82.13:4187: use of closed network connection Aug 7 23:54:20.126352653 (14h14m19.71569764s) [noise=true , address=104.36.82.12:4151 , piece_id=WME5KYWG2SHXNCORNMIXQGVSR56KCMIQ4VLSVC5TZFAFST7VPBXA, seg_pos={0 0}, upload=14754]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:37048->104.36.82.12:4151: use of closed network connection; write tcp 188.166.225.170:37048->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37048->104.36.82.12:4151: use of closed network connection Aug 7 23:54:20.126374925 (14h14m19.715719952s) [noise=true , address=104.36.82.12:4149 , piece_id=CLMPWWI6PYPQ6GKI2GKU2KYGRTGWH4A7RWD2BFG2EN2LQD4Z2XNA, seg_pos={0 0}, upload=14754]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: write tcp 188.166.225.170:53060->104.36.82.12:4149: use of closed network connection; write tcp 188.166.225.170:53060->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53060->104.36.82.12:4149: use of closed network connection Aug 7 23:54:20.126416585 (14h14m19.715761572s) [seg_pos={0 0}, upload=14754]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:54:20.126488342 (14h14m19.715833329s) [upload=14754]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:54:20.126492260 (14h14m19.715837247s) [upload=14754]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:20.421588191 (14h14m20.010933178s) [upload=14754]: Done uploading segments... Aug 7 23:54:20.421607058 (14h14m20.010952045s) [upload=14754]: Upload finished. err: <nil> Aug 7 23:54:46.372585852 (14h14m45.961930839s) [upload=14762]: Starting upload Aug 7 23:54:46.372635177 (14h14m45.961980164s) [upload=14762]: Done starting upload Aug 7 23:54:46.372803810 (14h14m45.962148787s) [upload=14762]: Uploading segments... Aug 7 23:54:46.372828878 (14h14m45.962173865s) [upload=14762]: Got next segment. Inline: false Aug 7 23:54:46.372849416 (14h14m45.962194403s) [upload=14762]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:46.372853104 (14h14m45.962198081s) [upload=14762]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:55:15.725296073 (14h15m15.31464106s) [upload=14762]: Next segment err: context deadline exceeded Aug 7 23:55:15.725460528 (14h15m15.314805515s) [upload=14762]: Done uploading segments... Aug 7 23:55:15.725496587 (14h15m15.314841574s) [upload=14762]: Upload finished. err: context deadline exceeded Duration: 30.001821816s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...X......X................................X................] duration: 2.673463157s FAIL (2025-08-07 23:55:10 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14406]: Done uploading segments... Aug 7 23:33:49.820540362 (13h53m49.409885379s) [upload=14406]: Upload finished. err: <nil> Aug 7 23:34:55.875843674 (13h54m55.465188661s) [upload=14425]: Starting upload Aug 7 23:34:55.875888460 (13h54m55.465233447s) [upload=14425]: Done starting upload Aug 7 23:34:55.875931803 (13h54m55.46527678s) [upload=14425]: Uploading segments... Aug 7 23:34:55.875961150 (13h54m55.465306137s) [upload=14425]: Got next segment. Inline: true Aug 7 23:34:55.875964626 (13h54m55.465309603s) [upload=14425]: Waiting for error group managing segments... Aug 7 23:34:55.875996888 (13h54m55.465341875s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:34:55.876001015 (13h54m55.465346002s) [upload=14425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:34:55.876003680 (13h54m55.465348667s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:34:56.158049155 (13h54m55.747394142s) [upload=14425]: Done uploading segments... Aug 7 23:34:56.158066439 (13h54m55.747411426s) [upload=14425]: Upload finished. err: <nil> Aug 7 23:35:36.771198141 (13h55m36.360543128s) [upload=14436]: Starting upload Aug 7 23:35:36.771239510 (13h55m36.360584487s) [upload=14436]: Done starting upload Aug 7 23:35:36.771282783 (13h55m36.36062776s) [upload=14436]: Uploading segments... Aug 7 23:35:36.771311258 (13h55m36.360656245s) [upload=14436]: Got next segment. Inline: true Aug 7 23:35:36.771315045 (13h55m36.360660032s) [upload=14436]: Waiting for error group managing segments... Aug 7 23:35:36.771344952 (13h55m36.360689939s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:35:36.771350051 (13h55m36.360695038s) [upload=14436]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:35:36.771352185 (13h55m36.360697172s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:35:37.066875398 (13h55m36.656220385s) [upload=14436]: Done uploading segments... Aug 7 23:35:37.066894275 (13h55m36.656239262s) [upload=14436]: Upload finished. err: <nil> Aug 7 23:36:11.909385420 (13h56m11.498730407s) [upload=14444]: Starting upload Aug 7 23:36:11.909445175 (13h56m11.498790172s) [upload=14444]: Done starting upload Aug 7 23:36:11.909503978 (13h56m11.498848985s) [upload=14444]: Uploading segments... Aug 7 23:36:11.909562349 (13h56m11.498907356s) [upload=14444]: Got next segment. Inline: true Aug 7 23:36:11.909568220 (13h56m11.498913207s) [upload=14444]: Waiting for error group managing segments... Aug 7 23:36:11.909608778 (13h56m11.498953765s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:11.909614439 (13h56m11.498959416s) [upload=14444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:11.909617855 (13h56m11.498962842s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:12.226741776 (13h56m11.816086753s) [upload=14444]: Done uploading segments... Aug 7 23:36:12.226761765 (13h56m11.816106752s) [upload=14444]: Upload finished. err: <nil> Aug 7 23:36:45.061365955 (13h56m44.650710942s) [upload=14452]: Starting upload Aug 7 23:36:45.061414287 (13h56m44.650759274s) [upload=14452]: Done starting upload Aug 7 23:36:45.061459804 (13h56m44.650804792s) [upload=14452]: Uploading segments... Aug 7 23:36:45.061489772 (13h56m44.650834759s) [upload=14452]: Got next segment. Inline: true Aug 7 23:36:45.061493318 (13h56m44.650838305s) [upload=14452]: Waiting for error group managing segments... Aug 7 23:36:45.061525611 (13h56m44.650870598s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:45.061529408 (13h56m44.650874395s) [upload=14452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:45.061531542 (13h56m44.650876529s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:45.340854790 (13h56m44.930199777s) [upload=14452]: Done uploading segments... Aug 7 23:36:45.340871481 (13h56m44.930216468s) [upload=14452]: Upload finished. err: <nil> Aug 7 23:37:27.165030871 (13h57m26.754375858s) [upload=14466]: Starting upload Aug 7 23:37:27.165168204 (13h57m26.754513201s) [upload=14466]: Done starting upload Aug 7 23:37:27.165252034 (13h57m26.754597061s) [upload=14466]: Uploading segments... Aug 7 23:37:27.165304224 (13h57m26.754649251s) [upload=14466]: Got next segment. Inline: true Aug 7 23:37:27.165311979 (13h57m26.754657006s) [upload=14466]: Waiting for error group managing segments... Aug 7 23:37:27.165356895 (13h57m26.754701872s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:37:27.165361664 (13h57m26.754706651s) [upload=14466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:37:27.165364449 (13h57m26.754709426s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:37:27.453150839 (13h57m27.042495816s) [upload=14466]: Done uploading segments... Aug 7 23:37:27.453172210 (13h57m27.042517218s) [upload=14466]: Upload finished. err: <nil> Aug 7 23:38:33.292631280 (13h58m32.881976267s) [upload=14482]: Starting upload Aug 7 23:38:33.292682438 (13h58m32.882027425s) [upload=14482]: Done starting upload Aug 7 23:38:33.292735750 (13h58m32.882080737s) [upload=14482]: Uploading segments... Aug 7 23:38:33.292771078 (13h58m32.882116065s) [upload=14482]: Got next segment. Inline: true Aug 7 23:38:33.292774524 (13h58m32.882119501s) [upload=14482]: Waiting for error group managing segments... Aug 7 23:38:33.292809602 (13h58m32.882154589s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:33.292814000 (13h58m32.882158987s) [upload=14482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:33.292816806 (13h58m32.882161793s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:33.652660690 (13h58m33.242005677s) [upload=14482]: Done uploading segments... Aug 7 23:38:33.652677502 (13h58m33.242022489s) [upload=14482]: Upload finished. err: <nil> Aug 7 23:38:56.173816507 (13h58m55.763161484s) [upload=14490]: Starting upload Aug 7 23:38:56.173867385 (13h58m55.763212372s) [upload=14490]: Done starting upload Aug 7 23:38:56.173920446 (13h58m55.763265433s) [upload=14490]: Uploading segments... Aug 7 23:38:56.173947418 (13h58m55.763292405s) [upload=14490]: Got next segment. Inline: true Aug 7 23:38:56.173951646 (13h58m55.763296633s) [upload=14490]: Waiting for error group managing segments... Aug 7 23:38:56.173986452 (13h58m55.763331439s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:56.173990099 (13h58m55.763335086s) [upload=14490]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:56.173992515 (13h58m55.763337492s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:56.442927417 (13h58m56.032272424s) [upload=14490]: Done uploading segments... Aug 7 23:38:56.442950000 (13h58m56.032294987s) [upload=14490]: Upload finished. err: <nil> Aug 7 23:39:51.382592315 (13h59m50.971937292s) [upload=14506]: Starting upload Aug 7 23:39:51.382637612 (13h59m50.971982599s) [upload=14506]: Done starting upload Aug 7 23:39:51.382703347 (13h59m50.972048324s) [upload=14506]: Uploading segments... Aug 7 23:39:51.382762962 (13h59m50.972107949s) [upload=14506]: Got next segment. Inline: true Aug 7 23:39:51.382767260 (13h59m50.972112247s) [upload=14506]: Waiting for error group managing segments... Aug 7 23:39:51.382800283 (13h59m50.97214527s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:39:51.382803920 (13h59m50.972148907s) [upload=14506]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:39:51.382806144 (13h59m50.972151131s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:39:51.664212669 (13h59m51.253557746s) [upload=14506]: Done uploading segments... Aug 7 23:39:51.664238148 (13h59m51.253583125s) [upload=14506]: Upload finished. err: <nil> Aug 7 23:40:08.314581416 (14h0m7.903926403s) [upload=14509]: Starting upload Aug 7 23:40:08.314628847 (14h0m7.903973834s) [upload=14509]: Done starting upload Aug 7 23:40:08.314675916 (14h0m7.904020903s) [upload=14509]: Uploading segments... Aug 7 23:40:08.314709010 (14h0m7.904053998s) [upload=14509]: Got next segment. Inline: true Aug 7 23:40:08.314712467 (14h0m7.904057454s) [upload=14509]: Waiting for error group managing segments... Aug 7 23:40:08.314744288 (14h0m7.904089275s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:40:08.314748175 (14h0m7.904093162s) [upload=14509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:40:08.314750489 (14h0m7.904095476s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:40:08.646371868 (14h0m8.235716855s) [upload=14509]: Done uploading segments... Aug 7 23:40:08.646389852 (14h0m8.235734829s) [upload=14509]: Upload finished. err: <nil> Aug 7 23:41:18.965192405 (14h1m18.554537392s) [upload=14528]: Starting upload Aug 7 23:41:18.965250195 (14h1m18.554595172s) [upload=14528]: Done starting upload Aug 7 23:41:18.965313206 (14h1m18.554658193s) [upload=14528]: Uploading segments... Aug 7 23:41:18.965345408 (14h1m18.554690395s) [upload=14528]: Got next segment. Inline: true Aug 7 23:41:18.965349346 (14h1m18.554694333s) [upload=14528]: Waiting for error group managing segments... Aug 7 23:41:18.965382669 (14h1m18.554727656s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:41:18.965386546 (14h1m18.554731523s) [upload=14528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:41:18.965389191 (14h1m18.554734178s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:19.261666757 (14h1m18.851011744s) [upload=14528]: Done uploading segments... Aug 7 23:41:19.261683329 (14h1m18.851028316s) [upload=14528]: Upload finished. err: <nil> Duration: 2.673488597s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.......XXXXXX.X.X.........................................XX] duration: 1m0.000565259s FAIL (2025-08-07 23:55:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000574336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 8 out of 60 is failed (threshold=3) [.......XXXX.............................................XXXX] duration: 30.000635219s FAIL (2025-08-07 23:54:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [upload=14630]: Upload finished. err: <nil> Aug 7 23:48:01.687516274 (14h8m1.276861261s) [upload=14647]: Starting upload Aug 7 23:48:01.687609412 (14h8m1.276954409s) [upload=14647]: Done starting upload Aug 7 23:48:01.687680398 (14h8m1.277025385s) [upload=14647]: Uploading segments... Aug 7 23:48:01.687725454 (14h8m1.277070441s) [upload=14647]: Got next segment. Inline: true Aug 7 23:48:01.687731175 (14h8m1.277076163s) [upload=14647]: Waiting for error group managing segments... Aug 7 23:48:01.687779207 (14h8m1.277124194s) [upload=14647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:01.687784667 (14h8m1.277129664s) [upload=14647]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:01.687788285 (14h8m1.277133282s) [upload=14647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:01.957323975 (14h8m1.546668972s) [upload=14647]: Done uploading segments... Aug 7 23:48:01.957343102 (14h8m1.546688089s) [upload=14647]: Upload finished. err: <nil> Aug 7 23:48:17.710466019 (14h8m17.299811006s) [upload=14652]: Starting upload Aug 7 23:48:17.710547815 (14h8m17.299892802s) [upload=14652]: Done starting upload Aug 7 23:48:17.710624521 (14h8m17.299969508s) [upload=14652]: Uploading segments... Aug 7 23:48:17.710729823 (14h8m17.30007481s) [upload=14652]: Got next segment. Inline: true Aug 7 23:48:17.710734742 (14h8m17.300079719s) [upload=14652]: Waiting for error group managing segments... Aug 7 23:48:17.710799587 (14h8m17.300144574s) [upload=14652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:17.710804295 (14h8m17.300149273s) [upload=14652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:17.710807071 (14h8m17.300152058s) [upload=14652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:17.976849249 (14h8m17.566194236s) [upload=14652]: Done uploading segments... Aug 7 23:48:17.976866912 (14h8m17.566211899s) [upload=14652]: Upload finished. err: <nil> Aug 7 23:49:05.340307173 (14h9m4.92965215s) [upload=14667]: Starting upload Aug 7 23:49:05.340383028 (14h9m4.929728015s) [upload=14667]: Done starting upload Aug 7 23:49:05.340707899 (14h9m4.930052887s) [upload=14667]: Uploading segments... Aug 7 23:49:05.340775519 (14h9m4.930120496s) [upload=14667]: Got next segment. Inline: true Aug 7 23:49:05.340781871 (14h9m4.930126858s) [upload=14667]: Waiting for error group managing segments... Aug 7 23:49:05.340825234 (14h9m4.930170221s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:05.340830374 (14h9m4.930175361s) [upload=14667]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:05.340833269 (14h9m4.930178246s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:05.817578876 (14h9m5.406923853s) [upload=14667]: Done uploading segments... Aug 7 23:49:05.817594757 (14h9m5.406939744s) [upload=14667]: Upload finished. err: <nil> Aug 7 23:49:28.341886774 (14h9m27.931231761s) [upload=14673]: Starting upload Aug 7 23:49:28.341998327 (14h9m27.931343334s) [upload=14673]: Done starting upload Aug 7 23:49:28.342069203 (14h9m27.9314142s) [upload=14673]: Uploading segments... Aug 7 23:49:28.342116133 (14h9m27.93146113s) [upload=14673]: Got next segment. Inline: true Aug 7 23:49:28.342121513 (14h9m27.93146651s) [upload=14673]: Waiting for error group managing segments... Aug 7 23:49:28.342171930 (14h9m27.931516927s) [upload=14673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:28.342178332 (14h9m27.931523319s) [upload=14673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:28.342182279 (14h9m27.931527256s) [upload=14673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:28.643582626 (14h9m28.232927603s) [upload=14673]: Done uploading segments... Aug 7 23:49:28.643602353 (14h9m28.23294734s) [upload=14673]: Upload finished. err: <nil> Aug 7 23:50:31.093627952 (14h10m30.682972939s) [upload=14692]: Starting upload Aug 7 23:50:31.093695632 (14h10m30.683040619s) [upload=14692]: Done starting upload Aug 7 23:50:31.093766667 (14h10m30.683111654s) [upload=14692]: Uploading segments... Aug 7 23:50:31.093857141 (14h10m30.683202128s) [upload=14692]: Got next segment. Inline: true Aug 7 23:50:31.093862081 (14h10m30.683207068s) [upload=14692]: Waiting for error group managing segments... Aug 7 23:50:31.093897037 (14h10m30.683242024s) [upload=14692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:31.093918809 (14h10m30.683263796s) [upload=14692]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:31.093921514 (14h10m30.683266501s) [upload=14692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:31.356040796 (14h10m30.945385783s) [upload=14692]: Done uploading segments... Aug 7 23:50:31.356106673 (14h10m30.94545166s) [upload=14692]: Upload finished. err: <nil> Aug 7 23:51:20.638658191 (14h11m20.228003178s) [upload=14707]: Starting upload Aug 7 23:51:20.638712213 (14h11m20.2280572s) [upload=14707]: Done starting upload Aug 7 23:51:20.638762700 (14h11m20.228107687s) [upload=14707]: Uploading segments... Aug 7 23:51:20.638799431 (14h11m20.228144418s) [upload=14707]: Got next segment. Inline: true Aug 7 23:51:20.638803058 (14h11m20.228148035s) [upload=14707]: Waiting for error group managing segments... Aug 7 23:51:20.638833967 (14h11m20.228178954s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:20.638838004 (14h11m20.228182991s) [upload=14707]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:20.638840399 (14h11m20.228185386s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:20.988587205 (14h11m20.577932192s) [upload=14707]: Done uploading segments... Aug 7 23:51:20.988605008 (14h11m20.577949985s) [upload=14707]: Upload finished. err: <nil> Aug 7 23:51:46.820025196 (14h11m46.409370183s) [upload=14715]: Starting upload Aug 7 23:51:46.820120348 (14h11m46.409465325s) [upload=14715]: Done starting upload Aug 7 23:51:46.820168981 (14h11m46.409513958s) [upload=14715]: Uploading segments... Aug 7 23:51:46.820463975 (14h11m46.409808972s) [upload=14715]: Got next segment. Inline: true Aug 7 23:51:46.820469005 (14h11m46.409813982s) [upload=14715]: Waiting for error group managing segments... Aug 7 23:51:46.820509402 (14h11m46.40985439s) [upload=14715]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:46.820513340 (14h11m46.409858327s) [upload=14715]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:46.820515594 (14h11m46.409860581s) [upload=14715]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:16.169845537 (14h12m15.759190524s) [upload=14715]: Done uploading segments... Aug 7 23:52:16.169877608 (14h12m15.759222595s) [upload=14715]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:52:54.182819305 (14h12m53.772164292s) [upload=14736]: Starting upload Aug 7 23:52:54.182879129 (14h12m53.772224116s) [upload=14736]: Done starting upload Aug 7 23:52:54.182920949 (14h12m53.772265926s) [upload=14736]: Uploading segments... Aug 7 23:52:54.182946839 (14h12m53.772291826s) [upload=14736]: Got next segment. Inline: true Aug 7 23:52:54.182950245 (14h12m53.772295222s) [upload=14736]: Waiting for error group managing segments... Aug 7 23:52:54.182983399 (14h12m53.772328386s) [upload=14736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:54.182987035 (14h12m53.772332022s) [upload=14736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:54.182989680 (14h12m53.772334667s) [upload=14736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:23.541227271 (14h13m23.130591846s) [upload=14736]: Done uploading segments... Aug 7 23:53:23.541269933 (14h13m23.13061492s) [upload=14736]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:53:24.412432118 (14h13m24.001777105s) [upload=14742]: Starting upload Aug 7 23:53:24.412511009 (14h13m24.001855996s) [upload=14742]: Done starting upload Aug 7 23:53:24.412569290 (14h13m24.001914277s) [upload=14742]: Uploading segments... Aug 7 23:53:24.412638803 (14h13m24.001983831s) [upload=14742]: Got next segment. Inline: true Aug 7 23:53:24.412644354 (14h13m24.001989341s) [upload=14742]: Waiting for error group managing segments... Aug 7 23:53:24.412680903 (14h13m24.002025891s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:24.412685442 (14h13m24.002030429s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:24.412688137 (14h13m24.002033114s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:53.772057432 (14h13m53.361402419s) [upload=14742]: Done uploading segments... Aug 7 23:53:53.772119881 (14h13m53.361464908s) [upload=14742]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:30.496202349 (14h14m30.085547336s) [upload=14761]: Starting upload Aug 7 23:54:30.496270259 (14h14m30.085615246s) [upload=14761]: Done starting upload Aug 7 23:54:30.496341565 (14h14m30.085686552s) [upload=14761]: Uploading segments... Aug 7 23:54:30.496378957 (14h14m30.085723944s) [upload=14761]: Got next segment. Inline: true Aug 7 23:54:30.496383836 (14h14m30.085728823s) [upload=14761]: Waiting for error group managing segments... Aug 7 23:54:30.496441346 (14h14m30.085786333s) [upload=14761]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:30.496446426 (14h14m30.085791413s) [upload=14761]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:30.496449551 (14h14m30.085794539s) [upload=14761]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:59.847423574 (14h14m59.436768561s) [upload=14761]: Done uploading segments... Aug 7 23:54:59.847456066 (14h14m59.436801043s) [upload=14761]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000653926s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 646.08829ms target: https://us1.storj.io/login FAIL (2025-08-07 23:54:59 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 646.110602ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X............................................XX] duration: 30.001432035s FAIL (2025-08-07 23:54:56 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.00145036s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.....................X..............X........X.........] duration: 6.901897354s FAIL (2025-08-07 23:54:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 408ba8db7a0d8af05b7d5f8336d5c5f7f4f0063d72c98ffee3c421a5da299777 != before d0b3d19e255585f56d510f04d387098fcb83db6017871f75c2e11245b6143455 ======LOGS====== ished. err: <nil> Aug 7 23:48:24.457116068 (14h8m24.046461055s) [upload=14655]: Starting upload Aug 7 23:48:24.457179188 (14h8m24.046524175s) [upload=14655]: Done starting upload Aug 7 23:48:24.457293146 (14h8m24.046638133s) [upload=14655]: Uploading segments... Aug 7 23:48:24.457654307 (14h8m24.046999294s) [upload=14655]: Got next segment. Inline: false Aug 7 23:48:24.457709964 (14h8m24.047054951s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:24.457715013 (14h8m24.04706s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:48:24.901153105 (14h8m24.490498082s) [seg_pos={0 0}, upload=14655]: Begin upload segment... Aug 7 23:48:24.901189916 (14h8m24.490534903s) [seg_pos={0 0}, upload=14655]: Done begin upload segment. Aug 7 23:48:24.901208150 (14h8m24.490553137s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901547320 (14h8m24.490892297s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901866269 (14h8m24.491211257s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901979486 (14h8m24.491324473s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902167115 (14h8m24.491512102s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902393378 (14h8m24.491738355s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902443794 (14h8m24.491788771s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902663996 (14h8m24.492008983s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902931768 (14h8m24.492276755s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903278531 (14h8m24.492623518s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903535794 (14h8m24.492880781s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903745445 (14h8m24.493090432s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903975075 (14h8m24.493320062s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904281360 (14h8m24.493626337s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904481643 (14h8m24.49382663s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904668631 (14h8m24.494013608s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904852633 (14h8m24.49419762s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.905040943 (14h8m24.49438593s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:26.003752552 (14h8m25.593097539s) [upload=14655]: Next returned nil segment Aug 7 23:48:26.003765798 (14h8m25.593110785s) [upload=14655]: Waiting for error group managing segments... Aug 7 23:48:26.282577597 (14h8m25.871922574s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.322820552 (14h8m25.912165529s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.366928523 (14h8m25.95627351s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.822974095 (14h8m26.412319082s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.895356812 (14h8m26.4847018s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.915282265 (14h8m26.504627252s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.929180326 (14h8m26.518525313s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.126611753 (14h8m26.71595674s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.271660348 (14h8m26.861005335s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.274412033 (14h8m26.86375702s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.336714535 (14h8m26.926059522s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.759468572 (14h8m27.348813559s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.904353465 (14h8m27.493698452s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960895364 (14h8m27.550240341s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960922416 (14h8m27.550267403s) [seg_pos={0 0}, upload=14655]: Segment reached optimal threshold of 14 pieces. Aug 7 23:48:27.961026686 (14h8m27.550371673s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961266504 (14h8m27.550611491s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961284479 (14h8m27.550629456s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: rpc: dial tcp 104.36.82.5:4869: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:48:27.961415680 (14h8m27.550760667s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC): protocol: context canceled; context canceled Aug 7 23:48:27.961429706 (14h8m27.550774693s) [seg_pos={0 0}, upload=14655]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:48:27.961507495 (14h8m27.550852482s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:48:27.961511613 (14h8m27.5508566s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:28.311226257 (14h8m27.900571244s) [upload=14655]: Done uploading segments... Aug 7 23:48:28.311246106 (14h8m27.900591083s) [upload=14655]: Upload finished. err: <nil> Duration: 6.90191663s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........XXXX.X..XX.......................................X..] duration: 6.066948771s FAIL (2025-08-07 23:54:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.066971686s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [...........X.....X.X...................................X....] duration: 2.389876225s FAIL (2025-08-07 23:54:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.389899829s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [...X...X..X.X............................................XX.] duration: 1.905809718s FAIL (2025-08-07 23:54:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.905834345s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.....................X..............X........X........] duration: 6.237285933s FAIL (2025-08-07 23:54:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 408ba8db7a0d8af05b7d5f8336d5c5f7f4f0063d72c98ffee3c421a5da299777 != before d0b3d19e255585f56d510f04d387098fcb83db6017871f75c2e11245b6143455 ======LOGS====== ished. err: <nil> Aug 7 23:48:24.457116068 (14h8m24.046461055s) [upload=14655]: Starting upload Aug 7 23:48:24.457179188 (14h8m24.046524175s) [upload=14655]: Done starting upload Aug 7 23:48:24.457293146 (14h8m24.046638133s) [upload=14655]: Uploading segments... Aug 7 23:48:24.457654307 (14h8m24.046999294s) [upload=14655]: Got next segment. Inline: false Aug 7 23:48:24.457709964 (14h8m24.047054951s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:24.457715013 (14h8m24.04706s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:48:24.901153105 (14h8m24.490498082s) [seg_pos={0 0}, upload=14655]: Begin upload segment... Aug 7 23:48:24.901189916 (14h8m24.490534903s) [seg_pos={0 0}, upload=14655]: Done begin upload segment. Aug 7 23:48:24.901208150 (14h8m24.490553137s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901547320 (14h8m24.490892297s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901866269 (14h8m24.491211257s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901979486 (14h8m24.491324473s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902167115 (14h8m24.491512102s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902393378 (14h8m24.491738355s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902443794 (14h8m24.491788771s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902663996 (14h8m24.492008983s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902931768 (14h8m24.492276755s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903278531 (14h8m24.492623518s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903535794 (14h8m24.492880781s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903745445 (14h8m24.493090432s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903975075 (14h8m24.493320062s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904281360 (14h8m24.493626337s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904481643 (14h8m24.49382663s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904668631 (14h8m24.494013608s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904852633 (14h8m24.49419762s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.905040943 (14h8m24.49438593s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:26.003752552 (14h8m25.593097539s) [upload=14655]: Next returned nil segment Aug 7 23:48:26.003765798 (14h8m25.593110785s) [upload=14655]: Waiting for error group managing segments... Aug 7 23:48:26.282577597 (14h8m25.871922574s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.322820552 (14h8m25.912165529s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.366928523 (14h8m25.95627351s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.822974095 (14h8m26.412319082s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.895356812 (14h8m26.4847018s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.915282265 (14h8m26.504627252s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.929180326 (14h8m26.518525313s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.126611753 (14h8m26.71595674s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.271660348 (14h8m26.861005335s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.274412033 (14h8m26.86375702s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.336714535 (14h8m26.926059522s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.759468572 (14h8m27.348813559s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.904353465 (14h8m27.493698452s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960895364 (14h8m27.550240341s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960922416 (14h8m27.550267403s) [seg_pos={0 0}, upload=14655]: Segment reached optimal threshold of 14 pieces. Aug 7 23:48:27.961026686 (14h8m27.550371673s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961266504 (14h8m27.550611491s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961284479 (14h8m27.550629456s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: rpc: dial tcp 104.36.82.5:4869: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:48:27.961415680 (14h8m27.550760667s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC): protocol: context canceled; context canceled Aug 7 23:48:27.961429706 (14h8m27.550774693s) [seg_pos={0 0}, upload=14655]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:48:27.961507495 (14h8m27.550852482s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:48:27.961511613 (14h8m27.5508566s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:28.311226257 (14h8m27.900571244s) [upload=14655]: Done uploading segments... Aug 7 23:48:28.311246106 (14h8m27.900591083s) [upload=14655]: Upload finished. err: <nil> Duration: 6.237306602s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X..X.XXXX.X............................................XX..] duration: 8.507895917s FAIL (2025-08-07 23:54:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 14h11m59.789927345s) [seg_pos={0 0}, upload=14721]: Done begin upload segment. Aug 7 23:52:00.200599942 (14h11m59.789944959s) [noise=true , address=77.90.31.19:4843 , piece_id=NVXWQQYUL3RSYHJZG53GE2JQMW7FU3HIUPGPJYYQAJGULOICBHEA, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.200609069 (14h11m59.789954056s) [noise=true , address=77.90.31.21:4879 , piece_id=CIOUHI4S3EYON4OJ5VMRK6E5QHBFVM3LDUUUPE7GNR27UJPXWXOQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.200700554 (14h11m59.790045551s) [noise=true , address=104.36.82.7:4910 , piece_id=H5P5BTNIZQVZ2OTW5TBWHQHC6PLJXNCLAS2RJNDJZIDIMNZT4FVQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.201562724 (14h11m59.790907721s) [noise=true , address=45.91.116.14:4191 , piece_id=CYG3IQ6A57K2BJJ7WBRN65RPEHKIHCRNV6FEC3NUGZNUEW3VMLPQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.202140499 (14h11m59.791485496s) [noise=true , address=77.90.31.18:4805 , piece_id=2Y55RTN4G7HSPAB734OGCQNHK7HFDSKGAYDBIQOGRGWB2JS7IWWQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.202565311 (14h11m59.791910298s) [noise=true , address=77.90.31.12:4590 , piece_id=FJBGEBSFJPFPVKZFJ6OVQQZGYQLE6TW3STY3ODGZQBEXUSYCUHHQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.202941321 (14h11m59.792286308s) [noise=true , address=45.91.116.8:4048 , piece_id=PFXTFJB64UCZPXY2MRJ3QNZYTBJPMGSCEINTZUEDAZ6BTLDMF4ZA, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.203062854 (14h11m59.792407861s) [noise=true , address=45.91.116.4:4938 , piece_id=POIEFCB5CBXZW3TKO3ZMT4ZEJJIWZHBUWOPXRLKBEGYB74IESQDQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.203542401 (14h11m59.792887378s) [noise=true , address=45.91.116.5:4297 , piece_id=CUMAL7XFD66GLBD4RAKP3SNBO4AIEYJXVTXSUNX3BQXUPQQ6S55A, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.203900947 (14h11m59.793245944s) [noise=true , address=77.90.31.14:4663 , piece_id=F6IYWOJJPLNENQ7KZ742UTPAAQIAPAWRSMLYULRZVJIS4NPW7UVA, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.204331041 (14h11m59.793676018s) [noise=true , address=77.90.31.19:4841 , piece_id=ZTUCRHKDBK4EGME3U2XXWMT36NISJKBJEYGNAUIIAAM53WBKVKZA, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.204730284 (14h11m59.794075271s) [noise=true , address=45.91.116.6:4975 , piece_id=XLM5WWXS4VTEPCK6NQB4EUOSFFRUVO4MLCQH3SF67G6FO2QJNARQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.204740444 (14h11m59.794085431s) [noise=true , address=45.91.116.12:4226 , piece_id=UVRNWUJCXD7Q2VK3A6T5QPNIC4PW2JD36W74VMGDWSEQV62NRB6A, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.205044235 (14h11m59.794389232s) [noise=true , address=45.91.116.11:4118 , piece_id=DUTNCBIML6HDKSKTHE2CPMU25RBJB2GPLHQU2TSTAKVSAXPUXXNQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.205322417 (14h11m59.794667394s) [noise=true , address=104.36.82.12:4150 , piece_id=D4TYX25GZDVSTSSKXAGVXVSGMZIKC5ZOSDEU6QTQ2CYMG5MT5IKQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.205847752 (14h11m59.795192749s) [noise=true , address=104.36.82.2:4732 , piece_id=EG6RUU2IRWRGSCPU25XV3ESK2LXY3TKO4MJPFUITSRWO3LBLAPLQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:02.068398504 (14h12m1.657743501s) [upload=14721]: Next returned nil segment Aug 7 23:52:02.068415778 (14h12m1.657760765s) [upload=14721]: Waiting for error group managing segments... Aug 7 23:52:02.367422114 (14h12m1.956767111s) [noise=true , address=104.36.82.12:4150 , piece_id=D4TYX25GZDVSTSSKXAGVXVSGMZIKC5ZOSDEU6QTQ2CYMG5MT5IKQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.422374943 (14h12m2.01171993s) [noise=true , address=77.90.31.19:4841 , piece_id=ZTUCRHKDBK4EGME3U2XXWMT36NISJKBJEYGNAUIIAAM53WBKVKZA, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.608759795 (14h12m2.198104782s) [noise=true , address=104.36.82.2:4732 , piece_id=EG6RUU2IRWRGSCPU25XV3ESK2LXY3TKO4MJPFUITSRWO3LBLAPLQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.639784323 (14h12m2.22912934s) [noise=true , address=77.90.31.19:4843 , piece_id=NVXWQQYUL3RSYHJZG53GE2JQMW7FU3HIUPGPJYYQAJGULOICBHEA, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.662434068 (14h12m2.251779055s) [noise=true , address=45.91.116.4:4938 , piece_id=POIEFCB5CBXZW3TKO3ZMT4ZEJJIWZHBUWOPXRLKBEGYB74IESQDQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.729318345 (14h12m2.318663333s) [noise=true , address=45.91.116.6:4975 , piece_id=XLM5WWXS4VTEPCK6NQB4EUOSFFRUVO4MLCQH3SF67G6FO2QJNARQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.774437961 (14h12m2.363782948s) [noise=true , address=45.91.116.8:4048 , piece_id=PFXTFJB64UCZPXY2MRJ3QNZYTBJPMGSCEINTZUEDAZ6BTLDMF4ZA, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.836024912 (14h12m2.425369899s) [noise=true , address=45.91.116.11:4118 , piece_id=DUTNCBIML6HDKSKTHE2CPMU25RBJB2GPLHQU2TSTAKVSAXPUXXNQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.847295246 (14h12m2.436640263s) [noise=true , address=104.36.82.7:4910 , piece_id=H5P5BTNIZQVZ2OTW5TBWHQHC6PLJXNCLAS2RJNDJZIDIMNZT4FVQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.848026284 (14h12m2.437371271s) [noise=true , address=77.90.31.12:4590 , piece_id=FJBGEBSFJPFPVKZFJ6OVQQZGYQLE6TW3STY3ODGZQBEXUSYCUHHQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.856271830 (14h12m2.445616818s) [noise=true , address=45.91.116.12:4226 , piece_id=UVRNWUJCXD7Q2VK3A6T5QPNIC4PW2JD36W74VMGDWSEQV62NRB6A, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.893735638 (14h12m2.483080626s) [noise=true , address=45.91.116.5:4297 , piece_id=CUMAL7XFD66GLBD4RAKP3SNBO4AIEYJXVTXSUNX3BQXUPQQ6S55A, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.938610556 (14h12m2.527955553s) [noise=true , address=77.90.31.14:4663 , piece_id=F6IYWOJJPLNENQ7KZ742UTPAAQIAPAWRSMLYULRZVJIS4NPW7UVA, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.938631105 (14h12m2.527976092s) [seg_pos={0 0}, upload=14721]: Segment reached optimal threshold of 13 pieces. Aug 7 23:52:02.938748910 (14h12m2.528093897s) [noise=true , address=77.90.31.21:4879 , piece_id=CIOUHI4S3EYON4OJ5VMRK6E5QHBFVM3LDUUUPE7GNR27UJPXWXOQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Aug 7 23:52:02.938875732 (14h12m2.528220729s) [noise=true , address=45.91.116.14:4191 , piece_id=CYG3IQ6A57K2BJJ7WBRN65RPEHKIHCRNV6FEC3NUGZNUEW3VMLPQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Aug 7 23:52:02.939590790 (14h12m2.528935778s) [noise=true , address=77.90.31.18:4805 , piece_id=2Y55RTN4G7HSPAB734OGCQNHK7HFDSKGAYDBIQOGRGWB2JS7IWWQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 7 23:52:02.939610088 (14h12m2.528955075s) [seg_pos={0 0}, upload=14721]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:52:02.939723344 (14h12m2.529068331s) [upload=14721]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:52:02.939730077 (14h12m2.529075064s) [upload=14721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:03.232136988 (14h12m2.821481985s) [upload=14721]: Done uploading segments... Aug 7 23:52:03.232187184 (14h12m2.821532171s) [upload=14721]: Upload finished. err: <nil> Aug 7 23:52:22.386769154 (14h12m21.976114141s) [upload=14726]: Starting upload Aug 7 23:52:22.386825732 (14h12m21.976170719s) [upload=14726]: Done starting upload Aug 7 23:52:22.386882942 (14h12m21.976227939s) [upload=14726]: Uploading segments... Aug 7 23:52:22.386993543 (14h12m21.97633853s) [upload=14726]: Got next segment. Inline: false Aug 7 23:52:22.387016918 (14h12m21.976361905s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:22.387020896 (14h12m21.976365883s) [upload=14726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:52:51.743802201 (14h12m51.333147178s) [upload=14726]: Next segment err: context deadline exceeded Aug 7 23:52:51.743938321 (14h12m51.333283298s) [upload=14726]: Done uploading segments... Aug 7 23:52:51.743958701 (14h12m51.333303688s) [upload=14726]: Upload finished. err: context deadline exceeded Aug 7 23:53:02.377428441 (14h13m1.966773458s) [upload=14738]: Starting upload Aug 7 23:53:02.377547469 (14h13m1.966892496s) [upload=14738]: Done starting upload Aug 7 23:53:02.377639475 (14h13m1.966984513s) [upload=14738]: Uploading segments... Aug 7 23:53:02.378016907 (14h13m1.967361894s) [upload=14738]: Got next segment. Inline: false Aug 7 23:53:02.378059268 (14h13m1.967404286s) [upload=14738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:02.378064859 (14h13m1.967409846s) [upload=14738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:53:31.738985270 (14h13m31.328330257s) [upload=14738]: Next segment err: context deadline exceeded Aug 7 23:53:31.739205060 (14h13m31.328550047s) [upload=14738]: Done uploading segments... Aug 7 23:53:31.739217965 (14h13m31.328562952s) [upload=14738]: Upload finished. err: context deadline exceeded Duration: 8.507929291s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [....X...X..X.X............................................XX] duration: 30.001055472s FAIL (2025-08-07 23:54:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001079749s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X............................................X] duration: 30.000885687s FAIL (2025-08-07 23:54:09 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000896948s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............XX.X.........................................X.] duration: 6.365710695s FAIL (2025-08-07 23:54:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 7 23:51:37.769341937 (14h11m37.358686924s) [upload=14711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:37.769345805 (14h11m37.358690792s) [upload=14711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:51:50.586530426 (14h11m50.175875413s) [seg_pos={0 0}, upload=14711]: Begin upload segment... Aug 7 23:51:50.586576043 (14h11m50.17592103s) [seg_pos={0 0}, upload=14711]: Done begin upload segment. Aug 7 23:51:50.586598596 (14h11m50.175943583s) [noise=true , address=104.36.82.39:4759 , piece_id=ANPBGKMOKTEN2WCRBXDVYSHB77XLVTJIAC352IXT4IZM2JUIDYIQ, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.586644183 (14h11m50.17598916s) [noise=true , address=104.36.82.59:4325 , piece_id=FNMNJTMVSQIKKNLUKGAROYTSB3RRCG2WMPZW7HKDACJTU2GPPJQQ, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.586929749 (14h11m50.176274736s) [noise=true , address=104.36.82.58:4297 , piece_id=CANGKWK5HHVQUMXE3KWK6VSQOUIFRBT5C2FPPM2ZF2QXS2HCWQVA, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.587125384 (14h11m50.176470371s) [noise=true , address=104.36.82.17:4356 , piece_id=RDH56NMNVQ622KZTPMUHYLCXWQNU3DI53RNCBDWLCIP5E3QNOBFA, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.587194957 (14h11m50.176539944s) [noise=true , address=104.36.82.38:4722 , piece_id=F3DAYUJIVWPPSV6MFFKA5X5V3MFND4JIXILTCVM35U65RZPAWXIQ, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.587442841 (14h11m50.176787828s) [noise=true , address=104.36.82.36:4632 , piece_id=Z42754BHVKQDMW43EHF43VEPENYR2VJPRUENBZ5PDULJYR5ZUPIA, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.587696797 (14h11m50.177041774s) [noise=true , address=45.91.116.13:4184 , piece_id=3DOEOH5P5FNP27SPLUTG4U57MBTO4LOOVNWBKR677PL5G7VMKHEA, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.587966323 (14h11m50.17731131s) [noise=true , address=104.36.82.43:4850 , piece_id=VASG5INVILP5QNBXQEOHV6LYMSMUWTUBSXYLWU35Q4RLNPMKEWLQ, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.588243934 (14h11m50.177588921s) [noise=true , address=104.36.82.8:4987 , piece_id=F4BZ42LTYEJ6TYTYCJD2T7QD53DQLPNXJ25PDCCTQIQE7HYBUB5Q, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.588256839 (14h11m50.177601816s) [noise=true , address=104.36.82.60:4346 , piece_id=Q5FY7LWMFTVTWIMJGXPAYCESLYMSJHULESXGDDV6CHXJJ75HTIUA, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.588513760 (14h11m50.177858737s) [noise=true , address=45.91.116.10:4085 , piece_id=7GIJL2F7TUC5DSSIMEP2HVK6WWYWUHVSME7EINPU63VU62K5PFTA, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.588794798 (14h11m50.178139785s) [noise=true , address=64.49.13.169:4707 , piece_id=B3RH6EARYZML23RUMYQORR45AIMNK44U3KZDJB4PJTBUA53MB3FQ, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.589254628 (14h11m50.178599615s) [noise=true , address=104.36.82.60:4349 , piece_id=QV6UT77TVSLEUEITJ2HNQTWTHPLSQNKIKEYCHCIY73YLFW6APFPQ, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.589373385 (14h11m50.178718372s) [noise=true , address=104.36.82.46:4506 , piece_id=TAET7WEONLENND3DPQTKAZ2VFHUSCUUDMJIUUP7MDGCWTLSINF3Q, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.589603716 (14h11m50.178948703s) [noise=true , address=79.127.205.233:4632 , piece_id=74CF4GICE4ASWV422BQ4GYCQBB5FMTI5LVQH5XPYTAMC5GNRIS4Q, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.589839036 (14h11m50.179184023s) [noise=true , address=64.49.13.154:4936 , piece_id=6NAZHC4EV4GQNQ64RRJQGUSMZK4VODN7NX5VEUJNVAJNTNUAOXYA, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.590037305 (14h11m50.179382292s) [noise=true , address=104.36.82.23:4466 , piece_id=KJGYJ5XJDCZ5TWGU2CCVUTHDDSBD2HVHGFGIK6KSN7YRSVOY7HUQ, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.590108441 (14h11m50.179453428s) [noise=true , address=64.49.13.155:4920 , piece_id=OELZWI5OFASJQIZU56CI54LB6MYYM7MCH2EPM3SANMVDMERRIO7A, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:51.690601519 (14h11m51.279946506s) [upload=14711]: Next returned nil segment Aug 7 23:51:51.690613823 (14h11m51.27995881s) [upload=14711]: Waiting for error group managing segments... Aug 7 23:51:51.988344770 (14h11m51.577689747s) [noise=true , address=79.127.205.233:4632 , piece_id=74CF4GICE4ASWV422BQ4GYCQBB5FMTI5LVQH5XPYTAMC5GNRIS4Q, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.005053097 (14h11m51.594398094s) [noise=true , address=64.49.13.169:4707 , piece_id=B3RH6EARYZML23RUMYQORR45AIMNK44U3KZDJB4PJTBUA53MB3FQ, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.014393979 (14h11m51.603738956s) [noise=true , address=64.49.13.155:4920 , piece_id=OELZWI5OFASJQIZU56CI54LB6MYYM7MCH2EPM3SANMVDMERRIO7A, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.040543139 (14h11m51.629888116s) [noise=true , address=64.49.13.154:4936 , piece_id=6NAZHC4EV4GQNQ64RRJQGUSMZK4VODN7NX5VEUJNVAJNTNUAOXYA, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.582382007 (14h11m52.171726984s) [noise=true , address=104.36.82.38:4722 , piece_id=F3DAYUJIVWPPSV6MFFKA5X5V3MFND4JIXILTCVM35U65RZPAWXIQ, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.586190804 (14h11m52.175535791s) [noise=true , address=104.36.82.59:4325 , piece_id=FNMNJTMVSQIKKNLUKGAROYTSB3RRCG2WMPZW7HKDACJTU2GPPJQQ, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.608210574 (14h11m52.197555561s) [noise=true , address=104.36.82.17:4356 , piece_id=RDH56NMNVQ622KZTPMUHYLCXWQNU3DI53RNCBDWLCIP5E3QNOBFA, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.636091307 (14h11m52.225436294s) [noise=true , address=104.36.82.39:4759 , piece_id=ANPBGKMOKTEN2WCRBXDVYSHB77XLVTJIAC352IXT4IZM2JUIDYIQ, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.679529656 (14h11m52.268874643s) [noise=true , address=104.36.82.8:4987 , piece_id=F4BZ42LTYEJ6TYTYCJD2T7QD53DQLPNXJ25PDCCTQIQE7HYBUB5Q, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.691210233 (14h11m52.28055521s) [noise=true , address=104.36.82.60:4346 , piece_id=Q5FY7LWMFTVTWIMJGXPAYCESLYMSJHULESXGDDV6CHXJJ75HTIUA, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.699406695 (14h11m52.288751682s) [noise=true , address=45.91.116.10:4085 , piece_id=7GIJL2F7TUC5DSSIMEP2HVK6WWYWUHVSME7EINPU63VU62K5PFTA, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.726791730 (14h11m52.316136707s) [noise=true , address=104.36.82.60:4349 , piece_id=QV6UT77TVSLEUEITJ2HNQTWTHPLSQNKIKEYCHCIY73YLFW6APFPQ, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.756815874 (14h11m52.346160862s) [noise=true , address=104.36.82.23:4466 , piece_id=KJGYJ5XJDCZ5TWGU2CCVUTHDDSBD2HVHGFGIK6KSN7YRSVOY7HUQ, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.758547257 (14h11m52.347892244s) [noise=true , address=45.91.116.13:4184 , piece_id=3DOEOH5P5FNP27SPLUTG4U57MBTO4LOOVNWBKR677PL5G7VMKHEA, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.758569329 (14h11m52.347914316s) [seg_pos={0 0}, upload=14711]: Segment reached optimal threshold of 14 pieces. Aug 7 23:51:52.758966068 (14h11m52.348311055s) [noise=true , address=104.36.82.58:4297 , piece_id=CANGKWK5HHVQUMXE3KWK6VSQOUIFRBT5C2FPPM2ZF2QXS2HCWQVA, seg_pos={0 0}, upload=14711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fVAGmLTkEZ2oauRnaMgyV9vvbRKZHj6eKW5vf2yZbPMDpRwfW): protocol: rpc: dial tcp 104.36.82.58:4297: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:51:52.759027385 (14h11m52.348372372s) [noise=true , address=104.36.82.36:4632 , piece_id=Z42754BHVKQDMW43EHF43VEPENYR2VJPRUENBZ5PDULJYR5ZUPIA, seg_pos={0 0}, upload=14711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC): protocol: rpc: dial tcp 104.36.82.36:4632: operation was canceled; context canceled Aug 7 23:51:52.759070098 (14h11m52.348415085s) [noise=true , address=104.36.82.46:4506 , piece_id=TAET7WEONLENND3DPQTKAZ2VFHUSCUUDMJIUUP7MDGCWTLSINF3Q, seg_pos={0 0}, upload=14711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9): protocol: write tcp 188.166.225.170:41220->104.36.82.46:4506: use of closed network connection; write tcp 188.166.225.170:41220->104.36.82.46:4506: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41220->104.36.82.46:4506: use of closed network connection Aug 7 23:51:52.759158386 (14h11m52.348503373s) [noise=true , address=104.36.82.43:4850 , piece_id=VASG5INVILP5QNBXQEOHV6LYMSMUWTUBSXYLWU35Q4RLNPMKEWLQ, seg_pos={0 0}, upload=14711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CCMafkZzgzS3tBh9SDoTS6tx1hNUfWqH56ebYzxLpcVnK5Cix): protocol: context canceled; context canceled Aug 7 23:51:52.759172873 (14h11m52.34851786s) [seg_pos={0 0}, upload=14711]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:51:52.759233579 (14h11m52.348578566s) [upload=14711]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:51:52.759237768 (14h11m52.348582755s) [upload=14711]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:03.198057543 (14h14m2.78740255s) [upload=14711]: Done uploading segments... Aug 7 23:54:03.198088251 (14h14m2.787433248s) [upload=14711]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.365726498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 6 out of 60 is failed (threshold=3) [.......XXXXX..............................................X.] duration: 1.905323515s FAIL (2025-08-07 23:54:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 87 (14h3m53.963331664s) [upload=14577]: Done uploading segments... Aug 7 23:43:54.374004261 (14h3m53.963349248s) [upload=14577]: Upload finished. err: <nil> Aug 7 23:45:02.926868636 (14h5m2.516213633s) [upload=14596]: Starting upload Aug 7 23:45:02.926953629 (14h5m2.516298616s) [upload=14596]: Done starting upload Aug 7 23:45:02.927035546 (14h5m2.516380543s) [upload=14596]: Uploading segments... Aug 7 23:45:02.927085210 (14h5m2.516430207s) [upload=14596]: Got next segment. Inline: true Aug 7 23:45:02.927091413 (14h5m2.51643641s) [upload=14596]: Waiting for error group managing segments... Aug 7 23:45:02.927144184 (14h5m2.516489181s) [upload=14596]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:45:02.927150366 (14h5m2.516495353s) [upload=14596]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:45:02.927153672 (14h5m2.516498659s) [upload=14596]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:45:03.205838598 (14h5m2.795183595s) [upload=14596]: Done uploading segments... Aug 7 23:45:03.205861432 (14h5m2.795206429s) [upload=14596]: Upload finished. err: <nil> Aug 7 23:45:57.100124831 (14h5m56.689469818s) [upload=14612]: Starting upload Aug 7 23:45:57.100374589 (14h5m56.689719576s) [upload=14612]: Done starting upload Aug 7 23:45:57.100874736 (14h5m56.690219723s) [upload=14612]: Uploading segments... Aug 7 23:45:57.100923860 (14h5m56.690268847s) [upload=14612]: Got next segment. Inline: true Aug 7 23:45:57.100930493 (14h5m56.69027548s) [upload=14612]: Waiting for error group managing segments... Aug 7 23:45:57.100984736 (14h5m56.690329723s) [upload=14612]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:45:57.100991540 (14h5m56.690336527s) [upload=14612]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:45:57.100995287 (14h5m56.690340274s) [upload=14612]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:45:57.379513894 (14h5m56.968858882s) [upload=14612]: Done uploading segments... Aug 7 23:45:57.379529645 (14h5m56.968874632s) [upload=14612]: Upload finished. err: <nil> Aug 7 23:46:23.861186317 (14h6m23.450531294s) [upload=14622]: Starting upload Aug 7 23:46:23.861235791 (14h6m23.450580778s) [upload=14622]: Done starting upload Aug 7 23:46:23.861288822 (14h6m23.450633809s) [upload=14622]: Uploading segments... Aug 7 23:46:23.861318689 (14h6m23.450663676s) [upload=14622]: Got next segment. Inline: true Aug 7 23:46:23.861322477 (14h6m23.450667465s) [upload=14622]: Waiting for error group managing segments... Aug 7 23:46:23.861354639 (14h6m23.450699616s) [upload=14622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:23.861358396 (14h6m23.450703383s) [upload=14622]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:23.861361011 (14h6m23.450705998s) [upload=14622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:24.134706832 (14h6m23.724051819s) [upload=14622]: Done uploading segments... Aug 7 23:46:24.134723955 (14h6m23.724068942s) [upload=14622]: Upload finished. err: <nil> Aug 7 23:46:58.581309303 (14h6m58.17065429s) [upload=14631]: Starting upload Aug 7 23:46:58.581357686 (14h6m58.170702673s) [upload=14631]: Done starting upload Aug 7 23:46:58.581420926 (14h6m58.170765954s) [upload=14631]: Uploading segments... Aug 7 23:46:58.581492674 (14h6m58.170837661s) [upload=14631]: Got next segment. Inline: true Aug 7 23:46:58.581497002 (14h6m58.170841989s) [upload=14631]: Waiting for error group managing segments... Aug 7 23:46:58.581529063 (14h6m58.17087405s) [upload=14631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:58.581533141 (14h6m58.170878128s) [upload=14631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:58.581535596 (14h6m58.170880573s) [upload=14631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:58.850274443 (14h6m58.43961943s) [upload=14631]: Done uploading segments... Aug 7 23:46:58.850297467 (14h6m58.439642474s) [upload=14631]: Upload finished. err: <nil> Aug 7 23:47:53.809895196 (14h7m53.399240183s) [upload=14646]: Starting upload Aug 7 23:47:53.809949169 (14h7m53.399294156s) [upload=14646]: Done starting upload Aug 7 23:47:53.809992792 (14h7m53.399337769s) [upload=14646]: Uploading segments... Aug 7 23:47:53.810021216 (14h7m53.399366203s) [upload=14646]: Got next segment. Inline: true Aug 7 23:47:53.810024943 (14h7m53.39936993s) [upload=14646]: Waiting for error group managing segments... Aug 7 23:47:53.810054902 (14h7m53.399399889s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:47:53.810058919 (14h7m53.399403896s) [upload=14646]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:47:53.810061294 (14h7m53.399406281s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:47:54.084012221 (14h7m53.673357208s) [upload=14646]: Done uploading segments... Aug 7 23:47:54.084030616 (14h7m53.673375603s) [upload=14646]: Upload finished. err: <nil> Aug 7 23:49:04.594851883 (14h9m4.18419687s) [upload=14666]: Starting upload Aug 7 23:49:04.594900156 (14h9m4.184245143s) [upload=14666]: Done starting upload Aug 7 23:49:04.594944311 (14h9m4.184289298s) [upload=14666]: Uploading segments... Aug 7 23:49:04.594984397 (14h9m4.184329384s) [upload=14666]: Got next segment. Inline: true Aug 7 23:49:04.594987814 (14h9m4.184332801s) [upload=14666]: Waiting for error group managing segments... Aug 7 23:49:04.595016589 (14h9m4.184361576s) [upload=14666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:04.595020496 (14h9m4.184365473s) [upload=14666]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:04.595022851 (14h9m4.184367828s) [upload=14666]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:04.868476128 (14h9m4.457821115s) [upload=14666]: Done uploading segments... Aug 7 23:49:04.868494303 (14h9m4.457839291s) [upload=14666]: Upload finished. err: <nil> Aug 7 23:50:10.977644727 (14h10m10.566989704s) [upload=14685]: Starting upload Aug 7 23:50:10.977704220 (14h10m10.567049207s) [upload=14685]: Done starting upload Aug 7 23:50:10.977750239 (14h10m10.567095226s) [upload=14685]: Uploading segments... Aug 7 23:50:10.977820534 (14h10m10.567165531s) [upload=14685]: Got next segment. Inline: true Aug 7 23:50:10.977824882 (14h10m10.567169869s) [upload=14685]: Waiting for error group managing segments... Aug 7 23:50:10.977858386 (14h10m10.567203373s) [upload=14685]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:10.977862343 (14h10m10.56720732s) [upload=14685]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:10.977864667 (14h10m10.567209644s) [upload=14685]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:11.274547609 (14h10m10.863892596s) [upload=14685]: Done uploading segments... Aug 7 23:50:11.274564251 (14h10m10.863909238s) [upload=14685]: Upload finished. err: <nil> Aug 7 23:50:53.129746032 (14h10m52.719091019s) [upload=14698]: Starting upload Aug 7 23:50:53.129817549 (14h10m52.719162526s) [upload=14698]: Done starting upload Aug 7 23:50:53.129869308 (14h10m52.719214285s) [upload=14698]: Uploading segments... Aug 7 23:50:53.129899716 (14h10m52.719244703s) [upload=14698]: Got next segment. Inline: true Aug 7 23:50:53.129903483 (14h10m52.719248471s) [upload=14698]: Waiting for error group managing segments... Aug 7 23:50:53.129935625 (14h10m52.719280612s) [upload=14698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:53.129939452 (14h10m52.719284439s) [upload=14698]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:53.129941756 (14h10m52.719286743s) [upload=14698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:53.612932053 (14h10m53.20227704s) [upload=14698]: Done uploading segments... Aug 7 23:50:53.612949836 (14h10m53.202294823s) [upload=14698]: Upload finished. err: <nil> Aug 7 23:51:55.415342045 (14h11m55.004687042s) [upload=14719]: Starting upload Aug 7 23:51:55.415462836 (14h11m55.004807824s) [upload=14719]: Done starting upload Aug 7 23:51:55.415539092 (14h11m55.004884089s) [upload=14719]: Uploading segments... Aug 7 23:51:55.415600830 (14h11m55.004945858s) [upload=14719]: Got next segment. Inline: true Aug 7 23:51:55.415607002 (14h11m55.004951989s) [upload=14719]: Waiting for error group managing segments... Aug 7 23:51:55.415668760 (14h11m55.005013747s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:55.415675372 (14h11m55.005020379s) [upload=14719]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:55.415679240 (14h11m55.005024237s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:55.825969401 (14h11m55.415314378s) [upload=14719]: Done uploading segments... Aug 7 23:51:55.825985652 (14h11m55.415330629s) [upload=14719]: Upload finished. err: <nil> Aug 7 23:52:50.979559156 (14h12m50.568904143s) [upload=14734]: Starting upload Aug 7 23:52:50.979632076 (14h12m50.568977063s) [upload=14734]: Done starting upload Aug 7 23:52:50.979699916 (14h12m50.569044903s) [upload=14734]: Uploading segments... Aug 7 23:52:50.979747066 (14h12m50.569092053s) [upload=14734]: Got next segment. Inline: true Aug 7 23:52:50.979751975 (14h12m50.569096962s) [upload=14734]: Waiting for error group managing segments... Aug 7 23:52:50.979789868 (14h12m50.569134845s) [upload=14734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:50.979794427 (14h12m50.569139414s) [upload=14734]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:50.979796861 (14h12m50.569141848s) [upload=14734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:50.328281597 (14h13m49.917626584s) [upload=14734]: Done uploading segments... Aug 7 23:53:50.328316053 (14h13m49.91766103s) [upload=14734]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.905351338s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [............X.....X.X...................................X...] duration: 2.255406918s FAIL (2025-08-07 23:54:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.255435683s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............XX.X.........................................X] duration: 2m26.077376422s FAIL (2025-08-07 23:54:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 7 23:51:37.769341937 (14h11m37.358686924s) [upload=14711]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:37.769345805 (14h11m37.358690792s) [upload=14711]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:51:50.586530426 (14h11m50.175875413s) [seg_pos={0 0}, upload=14711]: Begin upload segment... Aug 7 23:51:50.586576043 (14h11m50.17592103s) [seg_pos={0 0}, upload=14711]: Done begin upload segment. Aug 7 23:51:50.586598596 (14h11m50.175943583s) [noise=true , address=104.36.82.39:4759 , piece_id=ANPBGKMOKTEN2WCRBXDVYSHB77XLVTJIAC352IXT4IZM2JUIDYIQ, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.586644183 (14h11m50.17598916s) [noise=true , address=104.36.82.59:4325 , piece_id=FNMNJTMVSQIKKNLUKGAROYTSB3RRCG2WMPZW7HKDACJTU2GPPJQQ, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.586929749 (14h11m50.176274736s) [noise=true , address=104.36.82.58:4297 , piece_id=CANGKWK5HHVQUMXE3KWK6VSQOUIFRBT5C2FPPM2ZF2QXS2HCWQVA, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.587125384 (14h11m50.176470371s) [noise=true , address=104.36.82.17:4356 , piece_id=RDH56NMNVQ622KZTPMUHYLCXWQNU3DI53RNCBDWLCIP5E3QNOBFA, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.587194957 (14h11m50.176539944s) [noise=true , address=104.36.82.38:4722 , piece_id=F3DAYUJIVWPPSV6MFFKA5X5V3MFND4JIXILTCVM35U65RZPAWXIQ, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.587442841 (14h11m50.176787828s) [noise=true , address=104.36.82.36:4632 , piece_id=Z42754BHVKQDMW43EHF43VEPENYR2VJPRUENBZ5PDULJYR5ZUPIA, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.587696797 (14h11m50.177041774s) [noise=true , address=45.91.116.13:4184 , piece_id=3DOEOH5P5FNP27SPLUTG4U57MBTO4LOOVNWBKR677PL5G7VMKHEA, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.587966323 (14h11m50.17731131s) [noise=true , address=104.36.82.43:4850 , piece_id=VASG5INVILP5QNBXQEOHV6LYMSMUWTUBSXYLWU35Q4RLNPMKEWLQ, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.588243934 (14h11m50.177588921s) [noise=true , address=104.36.82.8:4987 , piece_id=F4BZ42LTYEJ6TYTYCJD2T7QD53DQLPNXJ25PDCCTQIQE7HYBUB5Q, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.588256839 (14h11m50.177601816s) [noise=true , address=104.36.82.60:4346 , piece_id=Q5FY7LWMFTVTWIMJGXPAYCESLYMSJHULESXGDDV6CHXJJ75HTIUA, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.588513760 (14h11m50.177858737s) [noise=true , address=45.91.116.10:4085 , piece_id=7GIJL2F7TUC5DSSIMEP2HVK6WWYWUHVSME7EINPU63VU62K5PFTA, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.588794798 (14h11m50.178139785s) [noise=true , address=64.49.13.169:4707 , piece_id=B3RH6EARYZML23RUMYQORR45AIMNK44U3KZDJB4PJTBUA53MB3FQ, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.589254628 (14h11m50.178599615s) [noise=true , address=104.36.82.60:4349 , piece_id=QV6UT77TVSLEUEITJ2HNQTWTHPLSQNKIKEYCHCIY73YLFW6APFPQ, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.589373385 (14h11m50.178718372s) [noise=true , address=104.36.82.46:4506 , piece_id=TAET7WEONLENND3DPQTKAZ2VFHUSCUUDMJIUUP7MDGCWTLSINF3Q, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.589603716 (14h11m50.178948703s) [noise=true , address=79.127.205.233:4632 , piece_id=74CF4GICE4ASWV422BQ4GYCQBB5FMTI5LVQH5XPYTAMC5GNRIS4Q, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.589839036 (14h11m50.179184023s) [noise=true , address=64.49.13.154:4936 , piece_id=6NAZHC4EV4GQNQ64RRJQGUSMZK4VODN7NX5VEUJNVAJNTNUAOXYA, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.590037305 (14h11m50.179382292s) [noise=true , address=104.36.82.23:4466 , piece_id=KJGYJ5XJDCZ5TWGU2CCVUTHDDSBD2HVHGFGIK6KSN7YRSVOY7HUQ, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:50.590108441 (14h11m50.179453428s) [noise=true , address=64.49.13.155:4920 , piece_id=OELZWI5OFASJQIZU56CI54LB6MYYM7MCH2EPM3SANMVDMERRIO7A, seg_pos={0 0}, upload=14711]: Uploading piece... Aug 7 23:51:51.690601519 (14h11m51.279946506s) [upload=14711]: Next returned nil segment Aug 7 23:51:51.690613823 (14h11m51.27995881s) [upload=14711]: Waiting for error group managing segments... Aug 7 23:51:51.988344770 (14h11m51.577689747s) [noise=true , address=79.127.205.233:4632 , piece_id=74CF4GICE4ASWV422BQ4GYCQBB5FMTI5LVQH5XPYTAMC5GNRIS4Q, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.005053097 (14h11m51.594398094s) [noise=true , address=64.49.13.169:4707 , piece_id=B3RH6EARYZML23RUMYQORR45AIMNK44U3KZDJB4PJTBUA53MB3FQ, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.014393979 (14h11m51.603738956s) [noise=true , address=64.49.13.155:4920 , piece_id=OELZWI5OFASJQIZU56CI54LB6MYYM7MCH2EPM3SANMVDMERRIO7A, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.040543139 (14h11m51.629888116s) [noise=true , address=64.49.13.154:4936 , piece_id=6NAZHC4EV4GQNQ64RRJQGUSMZK4VODN7NX5VEUJNVAJNTNUAOXYA, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.582382007 (14h11m52.171726984s) [noise=true , address=104.36.82.38:4722 , piece_id=F3DAYUJIVWPPSV6MFFKA5X5V3MFND4JIXILTCVM35U65RZPAWXIQ, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.586190804 (14h11m52.175535791s) [noise=true , address=104.36.82.59:4325 , piece_id=FNMNJTMVSQIKKNLUKGAROYTSB3RRCG2WMPZW7HKDACJTU2GPPJQQ, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.608210574 (14h11m52.197555561s) [noise=true , address=104.36.82.17:4356 , piece_id=RDH56NMNVQ622KZTPMUHYLCXWQNU3DI53RNCBDWLCIP5E3QNOBFA, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.636091307 (14h11m52.225436294s) [noise=true , address=104.36.82.39:4759 , piece_id=ANPBGKMOKTEN2WCRBXDVYSHB77XLVTJIAC352IXT4IZM2JUIDYIQ, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.679529656 (14h11m52.268874643s) [noise=true , address=104.36.82.8:4987 , piece_id=F4BZ42LTYEJ6TYTYCJD2T7QD53DQLPNXJ25PDCCTQIQE7HYBUB5Q, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.691210233 (14h11m52.28055521s) [noise=true , address=104.36.82.60:4346 , piece_id=Q5FY7LWMFTVTWIMJGXPAYCESLYMSJHULESXGDDV6CHXJJ75HTIUA, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.699406695 (14h11m52.288751682s) [noise=true , address=45.91.116.10:4085 , piece_id=7GIJL2F7TUC5DSSIMEP2HVK6WWYWUHVSME7EINPU63VU62K5PFTA, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.726791730 (14h11m52.316136707s) [noise=true , address=104.36.82.60:4349 , piece_id=QV6UT77TVSLEUEITJ2HNQTWTHPLSQNKIKEYCHCIY73YLFW6APFPQ, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.756815874 (14h11m52.346160862s) [noise=true , address=104.36.82.23:4466 , piece_id=KJGYJ5XJDCZ5TWGU2CCVUTHDDSBD2HVHGFGIK6KSN7YRSVOY7HUQ, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.758547257 (14h11m52.347892244s) [noise=true , address=45.91.116.13:4184 , piece_id=3DOEOH5P5FNP27SPLUTG4U57MBTO4LOOVNWBKR677PL5G7VMKHEA, seg_pos={0 0}, upload=14711]: Done uploading piece. err: <nil> Aug 7 23:51:52.758569329 (14h11m52.347914316s) [seg_pos={0 0}, upload=14711]: Segment reached optimal threshold of 14 pieces. Aug 7 23:51:52.758966068 (14h11m52.348311055s) [noise=true , address=104.36.82.58:4297 , piece_id=CANGKWK5HHVQUMXE3KWK6VSQOUIFRBT5C2FPPM2ZF2QXS2HCWQVA, seg_pos={0 0}, upload=14711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fVAGmLTkEZ2oauRnaMgyV9vvbRKZHj6eKW5vf2yZbPMDpRwfW): protocol: rpc: dial tcp 104.36.82.58:4297: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:51:52.759027385 (14h11m52.348372372s) [noise=true , address=104.36.82.36:4632 , piece_id=Z42754BHVKQDMW43EHF43VEPENYR2VJPRUENBZ5PDULJYR5ZUPIA, seg_pos={0 0}, upload=14711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC): protocol: rpc: dial tcp 104.36.82.36:4632: operation was canceled; context canceled Aug 7 23:51:52.759070098 (14h11m52.348415085s) [noise=true , address=104.36.82.46:4506 , piece_id=TAET7WEONLENND3DPQTKAZ2VFHUSCUUDMJIUUP7MDGCWTLSINF3Q, seg_pos={0 0}, upload=14711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9): protocol: write tcp 188.166.225.170:41220->104.36.82.46:4506: use of closed network connection; write tcp 188.166.225.170:41220->104.36.82.46:4506: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41220->104.36.82.46:4506: use of closed network connection Aug 7 23:51:52.759158386 (14h11m52.348503373s) [noise=true , address=104.36.82.43:4850 , piece_id=VASG5INVILP5QNBXQEOHV6LYMSMUWTUBSXYLWU35Q4RLNPMKEWLQ, seg_pos={0 0}, upload=14711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CCMafkZzgzS3tBh9SDoTS6tx1hNUfWqH56ebYzxLpcVnK5Cix): protocol: context canceled; context canceled Aug 7 23:51:52.759172873 (14h11m52.34851786s) [seg_pos={0 0}, upload=14711]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:51:52.759233579 (14h11m52.348578566s) [upload=14711]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:51:52.759237768 (14h11m52.348582755s) [upload=14711]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:03.198057543 (14h14m2.78740255s) [upload=14711]: Done uploading segments... Aug 7 23:54:03.198088251 (14h14m2.787433248s) [upload=14711]: Upload finished. err: metaclient: context deadline exceeded Duration: 2m26.077398504s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [........XXXXXX.X.X.........................................X] duration: 1m0.000474008s FAIL (2025-08-07 23:54:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000491864s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 648.819225ms target: https://us1.storj.io/login FAIL (2025-08-07 23:53:57 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 648.842418ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....X......X................................X...............] duration: 1.956371475s FAIL (2025-08-07 23:53:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14406]: Done uploading segments... Aug 7 23:33:49.820540362 (13h53m49.409885379s) [upload=14406]: Upload finished. err: <nil> Aug 7 23:34:55.875843674 (13h54m55.465188661s) [upload=14425]: Starting upload Aug 7 23:34:55.875888460 (13h54m55.465233447s) [upload=14425]: Done starting upload Aug 7 23:34:55.875931803 (13h54m55.46527678s) [upload=14425]: Uploading segments... Aug 7 23:34:55.875961150 (13h54m55.465306137s) [upload=14425]: Got next segment. Inline: true Aug 7 23:34:55.875964626 (13h54m55.465309603s) [upload=14425]: Waiting for error group managing segments... Aug 7 23:34:55.875996888 (13h54m55.465341875s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:34:55.876001015 (13h54m55.465346002s) [upload=14425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:34:55.876003680 (13h54m55.465348667s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:34:56.158049155 (13h54m55.747394142s) [upload=14425]: Done uploading segments... Aug 7 23:34:56.158066439 (13h54m55.747411426s) [upload=14425]: Upload finished. err: <nil> Aug 7 23:35:36.771198141 (13h55m36.360543128s) [upload=14436]: Starting upload Aug 7 23:35:36.771239510 (13h55m36.360584487s) [upload=14436]: Done starting upload Aug 7 23:35:36.771282783 (13h55m36.36062776s) [upload=14436]: Uploading segments... Aug 7 23:35:36.771311258 (13h55m36.360656245s) [upload=14436]: Got next segment. Inline: true Aug 7 23:35:36.771315045 (13h55m36.360660032s) [upload=14436]: Waiting for error group managing segments... Aug 7 23:35:36.771344952 (13h55m36.360689939s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:35:36.771350051 (13h55m36.360695038s) [upload=14436]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:35:36.771352185 (13h55m36.360697172s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:35:37.066875398 (13h55m36.656220385s) [upload=14436]: Done uploading segments... Aug 7 23:35:37.066894275 (13h55m36.656239262s) [upload=14436]: Upload finished. err: <nil> Aug 7 23:36:11.909385420 (13h56m11.498730407s) [upload=14444]: Starting upload Aug 7 23:36:11.909445175 (13h56m11.498790172s) [upload=14444]: Done starting upload Aug 7 23:36:11.909503978 (13h56m11.498848985s) [upload=14444]: Uploading segments... Aug 7 23:36:11.909562349 (13h56m11.498907356s) [upload=14444]: Got next segment. Inline: true Aug 7 23:36:11.909568220 (13h56m11.498913207s) [upload=14444]: Waiting for error group managing segments... Aug 7 23:36:11.909608778 (13h56m11.498953765s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:11.909614439 (13h56m11.498959416s) [upload=14444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:11.909617855 (13h56m11.498962842s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:12.226741776 (13h56m11.816086753s) [upload=14444]: Done uploading segments... Aug 7 23:36:12.226761765 (13h56m11.816106752s) [upload=14444]: Upload finished. err: <nil> Aug 7 23:36:45.061365955 (13h56m44.650710942s) [upload=14452]: Starting upload Aug 7 23:36:45.061414287 (13h56m44.650759274s) [upload=14452]: Done starting upload Aug 7 23:36:45.061459804 (13h56m44.650804792s) [upload=14452]: Uploading segments... Aug 7 23:36:45.061489772 (13h56m44.650834759s) [upload=14452]: Got next segment. Inline: true Aug 7 23:36:45.061493318 (13h56m44.650838305s) [upload=14452]: Waiting for error group managing segments... Aug 7 23:36:45.061525611 (13h56m44.650870598s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:45.061529408 (13h56m44.650874395s) [upload=14452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:45.061531542 (13h56m44.650876529s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:45.340854790 (13h56m44.930199777s) [upload=14452]: Done uploading segments... Aug 7 23:36:45.340871481 (13h56m44.930216468s) [upload=14452]: Upload finished. err: <nil> Aug 7 23:37:27.165030871 (13h57m26.754375858s) [upload=14466]: Starting upload Aug 7 23:37:27.165168204 (13h57m26.754513201s) [upload=14466]: Done starting upload Aug 7 23:37:27.165252034 (13h57m26.754597061s) [upload=14466]: Uploading segments... Aug 7 23:37:27.165304224 (13h57m26.754649251s) [upload=14466]: Got next segment. Inline: true Aug 7 23:37:27.165311979 (13h57m26.754657006s) [upload=14466]: Waiting for error group managing segments... Aug 7 23:37:27.165356895 (13h57m26.754701872s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:37:27.165361664 (13h57m26.754706651s) [upload=14466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:37:27.165364449 (13h57m26.754709426s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:37:27.453150839 (13h57m27.042495816s) [upload=14466]: Done uploading segments... Aug 7 23:37:27.453172210 (13h57m27.042517218s) [upload=14466]: Upload finished. err: <nil> Aug 7 23:38:33.292631280 (13h58m32.881976267s) [upload=14482]: Starting upload Aug 7 23:38:33.292682438 (13h58m32.882027425s) [upload=14482]: Done starting upload Aug 7 23:38:33.292735750 (13h58m32.882080737s) [upload=14482]: Uploading segments... Aug 7 23:38:33.292771078 (13h58m32.882116065s) [upload=14482]: Got next segment. Inline: true Aug 7 23:38:33.292774524 (13h58m32.882119501s) [upload=14482]: Waiting for error group managing segments... Aug 7 23:38:33.292809602 (13h58m32.882154589s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:33.292814000 (13h58m32.882158987s) [upload=14482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:33.292816806 (13h58m32.882161793s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:33.652660690 (13h58m33.242005677s) [upload=14482]: Done uploading segments... Aug 7 23:38:33.652677502 (13h58m33.242022489s) [upload=14482]: Upload finished. err: <nil> Aug 7 23:38:56.173816507 (13h58m55.763161484s) [upload=14490]: Starting upload Aug 7 23:38:56.173867385 (13h58m55.763212372s) [upload=14490]: Done starting upload Aug 7 23:38:56.173920446 (13h58m55.763265433s) [upload=14490]: Uploading segments... Aug 7 23:38:56.173947418 (13h58m55.763292405s) [upload=14490]: Got next segment. Inline: true Aug 7 23:38:56.173951646 (13h58m55.763296633s) [upload=14490]: Waiting for error group managing segments... Aug 7 23:38:56.173986452 (13h58m55.763331439s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:56.173990099 (13h58m55.763335086s) [upload=14490]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:56.173992515 (13h58m55.763337492s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:56.442927417 (13h58m56.032272424s) [upload=14490]: Done uploading segments... Aug 7 23:38:56.442950000 (13h58m56.032294987s) [upload=14490]: Upload finished. err: <nil> Aug 7 23:39:51.382592315 (13h59m50.971937292s) [upload=14506]: Starting upload Aug 7 23:39:51.382637612 (13h59m50.971982599s) [upload=14506]: Done starting upload Aug 7 23:39:51.382703347 (13h59m50.972048324s) [upload=14506]: Uploading segments... Aug 7 23:39:51.382762962 (13h59m50.972107949s) [upload=14506]: Got next segment. Inline: true Aug 7 23:39:51.382767260 (13h59m50.972112247s) [upload=14506]: Waiting for error group managing segments... Aug 7 23:39:51.382800283 (13h59m50.97214527s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:39:51.382803920 (13h59m50.972148907s) [upload=14506]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:39:51.382806144 (13h59m50.972151131s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:39:51.664212669 (13h59m51.253557746s) [upload=14506]: Done uploading segments... Aug 7 23:39:51.664238148 (13h59m51.253583125s) [upload=14506]: Upload finished. err: <nil> Aug 7 23:40:08.314581416 (14h0m7.903926403s) [upload=14509]: Starting upload Aug 7 23:40:08.314628847 (14h0m7.903973834s) [upload=14509]: Done starting upload Aug 7 23:40:08.314675916 (14h0m7.904020903s) [upload=14509]: Uploading segments... Aug 7 23:40:08.314709010 (14h0m7.904053998s) [upload=14509]: Got next segment. Inline: true Aug 7 23:40:08.314712467 (14h0m7.904057454s) [upload=14509]: Waiting for error group managing segments... Aug 7 23:40:08.314744288 (14h0m7.904089275s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:40:08.314748175 (14h0m7.904093162s) [upload=14509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:40:08.314750489 (14h0m7.904095476s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:40:08.646371868 (14h0m8.235716855s) [upload=14509]: Done uploading segments... Aug 7 23:40:08.646389852 (14h0m8.235734829s) [upload=14509]: Upload finished. err: <nil> Aug 7 23:41:18.965192405 (14h1m18.554537392s) [upload=14528]: Starting upload Aug 7 23:41:18.965250195 (14h1m18.554595172s) [upload=14528]: Done starting upload Aug 7 23:41:18.965313206 (14h1m18.554658193s) [upload=14528]: Uploading segments... Aug 7 23:41:18.965345408 (14h1m18.554690395s) [upload=14528]: Got next segment. Inline: true Aug 7 23:41:18.965349346 (14h1m18.554694333s) [upload=14528]: Waiting for error group managing segments... Aug 7 23:41:18.965382669 (14h1m18.554727656s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:41:18.965386546 (14h1m18.554731523s) [upload=14528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:41:18.965389191 (14h1m18.554734178s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:19.261666757 (14h1m18.851011744s) [upload=14528]: Done uploading segments... Aug 7 23:41:19.261683329 (14h1m18.851028316s) [upload=14528]: Upload finished. err: <nil> Duration: 1.95639989s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [........XXXX.............................................XXX] duration: 30.001292037s FAIL (2025-08-07 23:53:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 23:46:21.679092257 (14h6m21.268437234s) [upload=14621]: Upload finished. err: <nil> Aug 7 23:46:51.968618755 (14h6m51.557963762s) [upload=14630]: Starting upload Aug 7 23:46:51.968727564 (14h6m51.558072551s) [upload=14630]: Done starting upload Aug 7 23:46:51.968806966 (14h6m51.558151983s) [upload=14630]: Uploading segments... Aug 7 23:46:51.968857222 (14h6m51.558202209s) [upload=14630]: Got next segment. Inline: true Aug 7 23:46:51.968861520 (14h6m51.558206507s) [upload=14630]: Waiting for error group managing segments... Aug 7 23:46:51.968910524 (14h6m51.558255511s) [upload=14630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:51.968925081 (14h6m51.558270068s) [upload=14630]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:51.968929641 (14h6m51.558274638s) [upload=14630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:52.240213477 (14h6m51.829558464s) [upload=14630]: Done uploading segments... Aug 7 23:46:52.240231210 (14h6m51.829576197s) [upload=14630]: Upload finished. err: <nil> Aug 7 23:48:01.687516274 (14h8m1.276861261s) [upload=14647]: Starting upload Aug 7 23:48:01.687609412 (14h8m1.276954409s) [upload=14647]: Done starting upload Aug 7 23:48:01.687680398 (14h8m1.277025385s) [upload=14647]: Uploading segments... Aug 7 23:48:01.687725454 (14h8m1.277070441s) [upload=14647]: Got next segment. Inline: true Aug 7 23:48:01.687731175 (14h8m1.277076163s) [upload=14647]: Waiting for error group managing segments... Aug 7 23:48:01.687779207 (14h8m1.277124194s) [upload=14647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:01.687784667 (14h8m1.277129664s) [upload=14647]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:01.687788285 (14h8m1.277133282s) [upload=14647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:01.957323975 (14h8m1.546668972s) [upload=14647]: Done uploading segments... Aug 7 23:48:01.957343102 (14h8m1.546688089s) [upload=14647]: Upload finished. err: <nil> Aug 7 23:48:17.710466019 (14h8m17.299811006s) [upload=14652]: Starting upload Aug 7 23:48:17.710547815 (14h8m17.299892802s) [upload=14652]: Done starting upload Aug 7 23:48:17.710624521 (14h8m17.299969508s) [upload=14652]: Uploading segments... Aug 7 23:48:17.710729823 (14h8m17.30007481s) [upload=14652]: Got next segment. Inline: true Aug 7 23:48:17.710734742 (14h8m17.300079719s) [upload=14652]: Waiting for error group managing segments... Aug 7 23:48:17.710799587 (14h8m17.300144574s) [upload=14652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:17.710804295 (14h8m17.300149273s) [upload=14652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:17.710807071 (14h8m17.300152058s) [upload=14652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:17.976849249 (14h8m17.566194236s) [upload=14652]: Done uploading segments... Aug 7 23:48:17.976866912 (14h8m17.566211899s) [upload=14652]: Upload finished. err: <nil> Aug 7 23:49:05.340307173 (14h9m4.92965215s) [upload=14667]: Starting upload Aug 7 23:49:05.340383028 (14h9m4.929728015s) [upload=14667]: Done starting upload Aug 7 23:49:05.340707899 (14h9m4.930052887s) [upload=14667]: Uploading segments... Aug 7 23:49:05.340775519 (14h9m4.930120496s) [upload=14667]: Got next segment. Inline: true Aug 7 23:49:05.340781871 (14h9m4.930126858s) [upload=14667]: Waiting for error group managing segments... Aug 7 23:49:05.340825234 (14h9m4.930170221s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:05.340830374 (14h9m4.930175361s) [upload=14667]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:05.340833269 (14h9m4.930178246s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:05.817578876 (14h9m5.406923853s) [upload=14667]: Done uploading segments... Aug 7 23:49:05.817594757 (14h9m5.406939744s) [upload=14667]: Upload finished. err: <nil> Aug 7 23:49:28.341886774 (14h9m27.931231761s) [upload=14673]: Starting upload Aug 7 23:49:28.341998327 (14h9m27.931343334s) [upload=14673]: Done starting upload Aug 7 23:49:28.342069203 (14h9m27.9314142s) [upload=14673]: Uploading segments... Aug 7 23:49:28.342116133 (14h9m27.93146113s) [upload=14673]: Got next segment. Inline: true Aug 7 23:49:28.342121513 (14h9m27.93146651s) [upload=14673]: Waiting for error group managing segments... Aug 7 23:49:28.342171930 (14h9m27.931516927s) [upload=14673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:28.342178332 (14h9m27.931523319s) [upload=14673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:28.342182279 (14h9m27.931527256s) [upload=14673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:28.643582626 (14h9m28.232927603s) [upload=14673]: Done uploading segments... Aug 7 23:49:28.643602353 (14h9m28.23294734s) [upload=14673]: Upload finished. err: <nil> Aug 7 23:50:31.093627952 (14h10m30.682972939s) [upload=14692]: Starting upload Aug 7 23:50:31.093695632 (14h10m30.683040619s) [upload=14692]: Done starting upload Aug 7 23:50:31.093766667 (14h10m30.683111654s) [upload=14692]: Uploading segments... Aug 7 23:50:31.093857141 (14h10m30.683202128s) [upload=14692]: Got next segment. Inline: true Aug 7 23:50:31.093862081 (14h10m30.683207068s) [upload=14692]: Waiting for error group managing segments... Aug 7 23:50:31.093897037 (14h10m30.683242024s) [upload=14692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:31.093918809 (14h10m30.683263796s) [upload=14692]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:31.093921514 (14h10m30.683266501s) [upload=14692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:31.356040796 (14h10m30.945385783s) [upload=14692]: Done uploading segments... Aug 7 23:50:31.356106673 (14h10m30.94545166s) [upload=14692]: Upload finished. err: <nil> Aug 7 23:51:20.638658191 (14h11m20.228003178s) [upload=14707]: Starting upload Aug 7 23:51:20.638712213 (14h11m20.2280572s) [upload=14707]: Done starting upload Aug 7 23:51:20.638762700 (14h11m20.228107687s) [upload=14707]: Uploading segments... Aug 7 23:51:20.638799431 (14h11m20.228144418s) [upload=14707]: Got next segment. Inline: true Aug 7 23:51:20.638803058 (14h11m20.228148035s) [upload=14707]: Waiting for error group managing segments... Aug 7 23:51:20.638833967 (14h11m20.228178954s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:20.638838004 (14h11m20.228182991s) [upload=14707]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:20.638840399 (14h11m20.228185386s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:20.988587205 (14h11m20.577932192s) [upload=14707]: Done uploading segments... Aug 7 23:51:20.988605008 (14h11m20.577949985s) [upload=14707]: Upload finished. err: <nil> Aug 7 23:51:46.820025196 (14h11m46.409370183s) [upload=14715]: Starting upload Aug 7 23:51:46.820120348 (14h11m46.409465325s) [upload=14715]: Done starting upload Aug 7 23:51:46.820168981 (14h11m46.409513958s) [upload=14715]: Uploading segments... Aug 7 23:51:46.820463975 (14h11m46.409808972s) [upload=14715]: Got next segment. Inline: true Aug 7 23:51:46.820469005 (14h11m46.409813982s) [upload=14715]: Waiting for error group managing segments... Aug 7 23:51:46.820509402 (14h11m46.40985439s) [upload=14715]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:46.820513340 (14h11m46.409858327s) [upload=14715]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:46.820515594 (14h11m46.409860581s) [upload=14715]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:16.169845537 (14h12m15.759190524s) [upload=14715]: Done uploading segments... Aug 7 23:52:16.169877608 (14h12m15.759222595s) [upload=14715]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:52:54.182819305 (14h12m53.772164292s) [upload=14736]: Starting upload Aug 7 23:52:54.182879129 (14h12m53.772224116s) [upload=14736]: Done starting upload Aug 7 23:52:54.182920949 (14h12m53.772265926s) [upload=14736]: Uploading segments... Aug 7 23:52:54.182946839 (14h12m53.772291826s) [upload=14736]: Got next segment. Inline: true Aug 7 23:52:54.182950245 (14h12m53.772295222s) [upload=14736]: Waiting for error group managing segments... Aug 7 23:52:54.182983399 (14h12m53.772328386s) [upload=14736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:54.182987035 (14h12m53.772332022s) [upload=14736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:54.182989680 (14h12m53.772334667s) [upload=14736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:23.541227271 (14h13m23.130591846s) [upload=14736]: Done uploading segments... Aug 7 23:53:23.541269933 (14h13m23.13061492s) [upload=14736]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:53:24.412432118 (14h13m24.001777105s) [upload=14742]: Starting upload Aug 7 23:53:24.412511009 (14h13m24.001855996s) [upload=14742]: Done starting upload Aug 7 23:53:24.412569290 (14h13m24.001914277s) [upload=14742]: Uploading segments... Aug 7 23:53:24.412638803 (14h13m24.001983831s) [upload=14742]: Got next segment. Inline: true Aug 7 23:53:24.412644354 (14h13m24.001989341s) [upload=14742]: Waiting for error group managing segments... Aug 7 23:53:24.412680903 (14h13m24.002025891s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:24.412685442 (14h13m24.002030429s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:24.412688137 (14h13m24.002033114s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:53.772057432 (14h13m53.361402419s) [upload=14742]: Done uploading segments... Aug 7 23:53:53.772119881 (14h13m53.361464908s) [upload=14742]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001303119s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........XXXX.X..XX.......................................X.] duration: 6.27093775s FAIL (2025-08-07 23:53:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.270986454s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 6 out of 60 is failed (threshold=3) [........XXXXX..............................................X] duration: 1m0.001142325s FAIL (2025-08-07 23:53:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 87 (14h3m53.963331664s) [upload=14577]: Done uploading segments... Aug 7 23:43:54.374004261 (14h3m53.963349248s) [upload=14577]: Upload finished. err: <nil> Aug 7 23:45:02.926868636 (14h5m2.516213633s) [upload=14596]: Starting upload Aug 7 23:45:02.926953629 (14h5m2.516298616s) [upload=14596]: Done starting upload Aug 7 23:45:02.927035546 (14h5m2.516380543s) [upload=14596]: Uploading segments... Aug 7 23:45:02.927085210 (14h5m2.516430207s) [upload=14596]: Got next segment. Inline: true Aug 7 23:45:02.927091413 (14h5m2.51643641s) [upload=14596]: Waiting for error group managing segments... Aug 7 23:45:02.927144184 (14h5m2.516489181s) [upload=14596]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:45:02.927150366 (14h5m2.516495353s) [upload=14596]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:45:02.927153672 (14h5m2.516498659s) [upload=14596]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:45:03.205838598 (14h5m2.795183595s) [upload=14596]: Done uploading segments... Aug 7 23:45:03.205861432 (14h5m2.795206429s) [upload=14596]: Upload finished. err: <nil> Aug 7 23:45:57.100124831 (14h5m56.689469818s) [upload=14612]: Starting upload Aug 7 23:45:57.100374589 (14h5m56.689719576s) [upload=14612]: Done starting upload Aug 7 23:45:57.100874736 (14h5m56.690219723s) [upload=14612]: Uploading segments... Aug 7 23:45:57.100923860 (14h5m56.690268847s) [upload=14612]: Got next segment. Inline: true Aug 7 23:45:57.100930493 (14h5m56.69027548s) [upload=14612]: Waiting for error group managing segments... Aug 7 23:45:57.100984736 (14h5m56.690329723s) [upload=14612]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:45:57.100991540 (14h5m56.690336527s) [upload=14612]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:45:57.100995287 (14h5m56.690340274s) [upload=14612]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:45:57.379513894 (14h5m56.968858882s) [upload=14612]: Done uploading segments... Aug 7 23:45:57.379529645 (14h5m56.968874632s) [upload=14612]: Upload finished. err: <nil> Aug 7 23:46:23.861186317 (14h6m23.450531294s) [upload=14622]: Starting upload Aug 7 23:46:23.861235791 (14h6m23.450580778s) [upload=14622]: Done starting upload Aug 7 23:46:23.861288822 (14h6m23.450633809s) [upload=14622]: Uploading segments... Aug 7 23:46:23.861318689 (14h6m23.450663676s) [upload=14622]: Got next segment. Inline: true Aug 7 23:46:23.861322477 (14h6m23.450667465s) [upload=14622]: Waiting for error group managing segments... Aug 7 23:46:23.861354639 (14h6m23.450699616s) [upload=14622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:23.861358396 (14h6m23.450703383s) [upload=14622]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:23.861361011 (14h6m23.450705998s) [upload=14622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:24.134706832 (14h6m23.724051819s) [upload=14622]: Done uploading segments... Aug 7 23:46:24.134723955 (14h6m23.724068942s) [upload=14622]: Upload finished. err: <nil> Aug 7 23:46:58.581309303 (14h6m58.17065429s) [upload=14631]: Starting upload Aug 7 23:46:58.581357686 (14h6m58.170702673s) [upload=14631]: Done starting upload Aug 7 23:46:58.581420926 (14h6m58.170765954s) [upload=14631]: Uploading segments... Aug 7 23:46:58.581492674 (14h6m58.170837661s) [upload=14631]: Got next segment. Inline: true Aug 7 23:46:58.581497002 (14h6m58.170841989s) [upload=14631]: Waiting for error group managing segments... Aug 7 23:46:58.581529063 (14h6m58.17087405s) [upload=14631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:58.581533141 (14h6m58.170878128s) [upload=14631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:58.581535596 (14h6m58.170880573s) [upload=14631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:58.850274443 (14h6m58.43961943s) [upload=14631]: Done uploading segments... Aug 7 23:46:58.850297467 (14h6m58.439642474s) [upload=14631]: Upload finished. err: <nil> Aug 7 23:47:53.809895196 (14h7m53.399240183s) [upload=14646]: Starting upload Aug 7 23:47:53.809949169 (14h7m53.399294156s) [upload=14646]: Done starting upload Aug 7 23:47:53.809992792 (14h7m53.399337769s) [upload=14646]: Uploading segments... Aug 7 23:47:53.810021216 (14h7m53.399366203s) [upload=14646]: Got next segment. Inline: true Aug 7 23:47:53.810024943 (14h7m53.39936993s) [upload=14646]: Waiting for error group managing segments... Aug 7 23:47:53.810054902 (14h7m53.399399889s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:47:53.810058919 (14h7m53.399403896s) [upload=14646]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:47:53.810061294 (14h7m53.399406281s) [upload=14646]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:47:54.084012221 (14h7m53.673357208s) [upload=14646]: Done uploading segments... Aug 7 23:47:54.084030616 (14h7m53.673375603s) [upload=14646]: Upload finished. err: <nil> Aug 7 23:49:04.594851883 (14h9m4.18419687s) [upload=14666]: Starting upload Aug 7 23:49:04.594900156 (14h9m4.184245143s) [upload=14666]: Done starting upload Aug 7 23:49:04.594944311 (14h9m4.184289298s) [upload=14666]: Uploading segments... Aug 7 23:49:04.594984397 (14h9m4.184329384s) [upload=14666]: Got next segment. Inline: true Aug 7 23:49:04.594987814 (14h9m4.184332801s) [upload=14666]: Waiting for error group managing segments... Aug 7 23:49:04.595016589 (14h9m4.184361576s) [upload=14666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:04.595020496 (14h9m4.184365473s) [upload=14666]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:04.595022851 (14h9m4.184367828s) [upload=14666]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:04.868476128 (14h9m4.457821115s) [upload=14666]: Done uploading segments... Aug 7 23:49:04.868494303 (14h9m4.457839291s) [upload=14666]: Upload finished. err: <nil> Aug 7 23:50:10.977644727 (14h10m10.566989704s) [upload=14685]: Starting upload Aug 7 23:50:10.977704220 (14h10m10.567049207s) [upload=14685]: Done starting upload Aug 7 23:50:10.977750239 (14h10m10.567095226s) [upload=14685]: Uploading segments... Aug 7 23:50:10.977820534 (14h10m10.567165531s) [upload=14685]: Got next segment. Inline: true Aug 7 23:50:10.977824882 (14h10m10.567169869s) [upload=14685]: Waiting for error group managing segments... Aug 7 23:50:10.977858386 (14h10m10.567203373s) [upload=14685]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:10.977862343 (14h10m10.56720732s) [upload=14685]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:10.977864667 (14h10m10.567209644s) [upload=14685]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:11.274547609 (14h10m10.863892596s) [upload=14685]: Done uploading segments... Aug 7 23:50:11.274564251 (14h10m10.863909238s) [upload=14685]: Upload finished. err: <nil> Aug 7 23:50:53.129746032 (14h10m52.719091019s) [upload=14698]: Starting upload Aug 7 23:50:53.129817549 (14h10m52.719162526s) [upload=14698]: Done starting upload Aug 7 23:50:53.129869308 (14h10m52.719214285s) [upload=14698]: Uploading segments... Aug 7 23:50:53.129899716 (14h10m52.719244703s) [upload=14698]: Got next segment. Inline: true Aug 7 23:50:53.129903483 (14h10m52.719248471s) [upload=14698]: Waiting for error group managing segments... Aug 7 23:50:53.129935625 (14h10m52.719280612s) [upload=14698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:53.129939452 (14h10m52.719284439s) [upload=14698]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:53.129941756 (14h10m52.719286743s) [upload=14698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:53.612932053 (14h10m53.20227704s) [upload=14698]: Done uploading segments... Aug 7 23:50:53.612949836 (14h10m53.202294823s) [upload=14698]: Upload finished. err: <nil> Aug 7 23:51:55.415342045 (14h11m55.004687042s) [upload=14719]: Starting upload Aug 7 23:51:55.415462836 (14h11m55.004807824s) [upload=14719]: Done starting upload Aug 7 23:51:55.415539092 (14h11m55.004884089s) [upload=14719]: Uploading segments... Aug 7 23:51:55.415600830 (14h11m55.004945858s) [upload=14719]: Got next segment. Inline: true Aug 7 23:51:55.415607002 (14h11m55.004951989s) [upload=14719]: Waiting for error group managing segments... Aug 7 23:51:55.415668760 (14h11m55.005013747s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:55.415675372 (14h11m55.005020379s) [upload=14719]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:55.415679240 (14h11m55.005024237s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:55.825969401 (14h11m55.415314378s) [upload=14719]: Done uploading segments... Aug 7 23:51:55.825985652 (14h11m55.415330629s) [upload=14719]: Upload finished. err: <nil> Aug 7 23:52:50.979559156 (14h12m50.568904143s) [upload=14734]: Starting upload Aug 7 23:52:50.979632076 (14h12m50.568977063s) [upload=14734]: Done starting upload Aug 7 23:52:50.979699916 (14h12m50.569044903s) [upload=14734]: Uploading segments... Aug 7 23:52:50.979747066 (14h12m50.569092053s) [upload=14734]: Got next segment. Inline: true Aug 7 23:52:50.979751975 (14h12m50.569096962s) [upload=14734]: Waiting for error group managing segments... Aug 7 23:52:50.979789868 (14h12m50.569134845s) [upload=14734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:50.979794427 (14h12m50.569139414s) [upload=14734]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:50.979796861 (14h12m50.569141848s) [upload=14734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:50.328281597 (14h13m49.917626584s) [upload=14734]: Done uploading segments... Aug 7 23:53:50.328316053 (14h13m49.91766103s) [upload=14734]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001155411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....X...X..X.X............................................X] duration: 30.000498676s FAIL (2025-08-07 23:53:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000520157s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.............X.....X.X...................................X..] duration: 2.480684716s FAIL (2025-08-07 23:53:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.480935305s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X..X.XXXX.X............................................XX.] duration: 6.594175906s FAIL (2025-08-07 23:53:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 14h11m59.789927345s) [seg_pos={0 0}, upload=14721]: Done begin upload segment. Aug 7 23:52:00.200599942 (14h11m59.789944959s) [noise=true , address=77.90.31.19:4843 , piece_id=NVXWQQYUL3RSYHJZG53GE2JQMW7FU3HIUPGPJYYQAJGULOICBHEA, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.200609069 (14h11m59.789954056s) [noise=true , address=77.90.31.21:4879 , piece_id=CIOUHI4S3EYON4OJ5VMRK6E5QHBFVM3LDUUUPE7GNR27UJPXWXOQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.200700554 (14h11m59.790045551s) [noise=true , address=104.36.82.7:4910 , piece_id=H5P5BTNIZQVZ2OTW5TBWHQHC6PLJXNCLAS2RJNDJZIDIMNZT4FVQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.201562724 (14h11m59.790907721s) [noise=true , address=45.91.116.14:4191 , piece_id=CYG3IQ6A57K2BJJ7WBRN65RPEHKIHCRNV6FEC3NUGZNUEW3VMLPQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.202140499 (14h11m59.791485496s) [noise=true , address=77.90.31.18:4805 , piece_id=2Y55RTN4G7HSPAB734OGCQNHK7HFDSKGAYDBIQOGRGWB2JS7IWWQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.202565311 (14h11m59.791910298s) [noise=true , address=77.90.31.12:4590 , piece_id=FJBGEBSFJPFPVKZFJ6OVQQZGYQLE6TW3STY3ODGZQBEXUSYCUHHQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.202941321 (14h11m59.792286308s) [noise=true , address=45.91.116.8:4048 , piece_id=PFXTFJB64UCZPXY2MRJ3QNZYTBJPMGSCEINTZUEDAZ6BTLDMF4ZA, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.203062854 (14h11m59.792407861s) [noise=true , address=45.91.116.4:4938 , piece_id=POIEFCB5CBXZW3TKO3ZMT4ZEJJIWZHBUWOPXRLKBEGYB74IESQDQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.203542401 (14h11m59.792887378s) [noise=true , address=45.91.116.5:4297 , piece_id=CUMAL7XFD66GLBD4RAKP3SNBO4AIEYJXVTXSUNX3BQXUPQQ6S55A, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.203900947 (14h11m59.793245944s) [noise=true , address=77.90.31.14:4663 , piece_id=F6IYWOJJPLNENQ7KZ742UTPAAQIAPAWRSMLYULRZVJIS4NPW7UVA, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.204331041 (14h11m59.793676018s) [noise=true , address=77.90.31.19:4841 , piece_id=ZTUCRHKDBK4EGME3U2XXWMT36NISJKBJEYGNAUIIAAM53WBKVKZA, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.204730284 (14h11m59.794075271s) [noise=true , address=45.91.116.6:4975 , piece_id=XLM5WWXS4VTEPCK6NQB4EUOSFFRUVO4MLCQH3SF67G6FO2QJNARQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.204740444 (14h11m59.794085431s) [noise=true , address=45.91.116.12:4226 , piece_id=UVRNWUJCXD7Q2VK3A6T5QPNIC4PW2JD36W74VMGDWSEQV62NRB6A, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.205044235 (14h11m59.794389232s) [noise=true , address=45.91.116.11:4118 , piece_id=DUTNCBIML6HDKSKTHE2CPMU25RBJB2GPLHQU2TSTAKVSAXPUXXNQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.205322417 (14h11m59.794667394s) [noise=true , address=104.36.82.12:4150 , piece_id=D4TYX25GZDVSTSSKXAGVXVSGMZIKC5ZOSDEU6QTQ2CYMG5MT5IKQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.205847752 (14h11m59.795192749s) [noise=true , address=104.36.82.2:4732 , piece_id=EG6RUU2IRWRGSCPU25XV3ESK2LXY3TKO4MJPFUITSRWO3LBLAPLQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:02.068398504 (14h12m1.657743501s) [upload=14721]: Next returned nil segment Aug 7 23:52:02.068415778 (14h12m1.657760765s) [upload=14721]: Waiting for error group managing segments... Aug 7 23:52:02.367422114 (14h12m1.956767111s) [noise=true , address=104.36.82.12:4150 , piece_id=D4TYX25GZDVSTSSKXAGVXVSGMZIKC5ZOSDEU6QTQ2CYMG5MT5IKQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.422374943 (14h12m2.01171993s) [noise=true , address=77.90.31.19:4841 , piece_id=ZTUCRHKDBK4EGME3U2XXWMT36NISJKBJEYGNAUIIAAM53WBKVKZA, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.608759795 (14h12m2.198104782s) [noise=true , address=104.36.82.2:4732 , piece_id=EG6RUU2IRWRGSCPU25XV3ESK2LXY3TKO4MJPFUITSRWO3LBLAPLQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.639784323 (14h12m2.22912934s) [noise=true , address=77.90.31.19:4843 , piece_id=NVXWQQYUL3RSYHJZG53GE2JQMW7FU3HIUPGPJYYQAJGULOICBHEA, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.662434068 (14h12m2.251779055s) [noise=true , address=45.91.116.4:4938 , piece_id=POIEFCB5CBXZW3TKO3ZMT4ZEJJIWZHBUWOPXRLKBEGYB74IESQDQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.729318345 (14h12m2.318663333s) [noise=true , address=45.91.116.6:4975 , piece_id=XLM5WWXS4VTEPCK6NQB4EUOSFFRUVO4MLCQH3SF67G6FO2QJNARQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.774437961 (14h12m2.363782948s) [noise=true , address=45.91.116.8:4048 , piece_id=PFXTFJB64UCZPXY2MRJ3QNZYTBJPMGSCEINTZUEDAZ6BTLDMF4ZA, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.836024912 (14h12m2.425369899s) [noise=true , address=45.91.116.11:4118 , piece_id=DUTNCBIML6HDKSKTHE2CPMU25RBJB2GPLHQU2TSTAKVSAXPUXXNQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.847295246 (14h12m2.436640263s) [noise=true , address=104.36.82.7:4910 , piece_id=H5P5BTNIZQVZ2OTW5TBWHQHC6PLJXNCLAS2RJNDJZIDIMNZT4FVQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.848026284 (14h12m2.437371271s) [noise=true , address=77.90.31.12:4590 , piece_id=FJBGEBSFJPFPVKZFJ6OVQQZGYQLE6TW3STY3ODGZQBEXUSYCUHHQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.856271830 (14h12m2.445616818s) [noise=true , address=45.91.116.12:4226 , piece_id=UVRNWUJCXD7Q2VK3A6T5QPNIC4PW2JD36W74VMGDWSEQV62NRB6A, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.893735638 (14h12m2.483080626s) [noise=true , address=45.91.116.5:4297 , piece_id=CUMAL7XFD66GLBD4RAKP3SNBO4AIEYJXVTXSUNX3BQXUPQQ6S55A, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.938610556 (14h12m2.527955553s) [noise=true , address=77.90.31.14:4663 , piece_id=F6IYWOJJPLNENQ7KZ742UTPAAQIAPAWRSMLYULRZVJIS4NPW7UVA, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.938631105 (14h12m2.527976092s) [seg_pos={0 0}, upload=14721]: Segment reached optimal threshold of 13 pieces. Aug 7 23:52:02.938748910 (14h12m2.528093897s) [noise=true , address=77.90.31.21:4879 , piece_id=CIOUHI4S3EYON4OJ5VMRK6E5QHBFVM3LDUUUPE7GNR27UJPXWXOQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Aug 7 23:52:02.938875732 (14h12m2.528220729s) [noise=true , address=45.91.116.14:4191 , piece_id=CYG3IQ6A57K2BJJ7WBRN65RPEHKIHCRNV6FEC3NUGZNUEW3VMLPQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Aug 7 23:52:02.939590790 (14h12m2.528935778s) [noise=true , address=77.90.31.18:4805 , piece_id=2Y55RTN4G7HSPAB734OGCQNHK7HFDSKGAYDBIQOGRGWB2JS7IWWQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 7 23:52:02.939610088 (14h12m2.528955075s) [seg_pos={0 0}, upload=14721]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:52:02.939723344 (14h12m2.529068331s) [upload=14721]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:52:02.939730077 (14h12m2.529075064s) [upload=14721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:03.232136988 (14h12m2.821481985s) [upload=14721]: Done uploading segments... Aug 7 23:52:03.232187184 (14h12m2.821532171s) [upload=14721]: Upload finished. err: <nil> Aug 7 23:52:22.386769154 (14h12m21.976114141s) [upload=14726]: Starting upload Aug 7 23:52:22.386825732 (14h12m21.976170719s) [upload=14726]: Done starting upload Aug 7 23:52:22.386882942 (14h12m21.976227939s) [upload=14726]: Uploading segments... Aug 7 23:52:22.386993543 (14h12m21.97633853s) [upload=14726]: Got next segment. Inline: false Aug 7 23:52:22.387016918 (14h12m21.976361905s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:22.387020896 (14h12m21.976365883s) [upload=14726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:52:51.743802201 (14h12m51.333147178s) [upload=14726]: Next segment err: context deadline exceeded Aug 7 23:52:51.743938321 (14h12m51.333283298s) [upload=14726]: Done uploading segments... Aug 7 23:52:51.743958701 (14h12m51.333303688s) [upload=14726]: Upload finished. err: context deadline exceeded Aug 7 23:53:02.377428441 (14h13m1.966773458s) [upload=14738]: Starting upload Aug 7 23:53:02.377547469 (14h13m1.966892496s) [upload=14738]: Done starting upload Aug 7 23:53:02.377639475 (14h13m1.966984513s) [upload=14738]: Uploading segments... Aug 7 23:53:02.378016907 (14h13m1.967361894s) [upload=14738]: Got next segment. Inline: false Aug 7 23:53:02.378059268 (14h13m1.967404286s) [upload=14738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:02.378064859 (14h13m1.967409846s) [upload=14738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:53:31.738985270 (14h13m31.328330257s) [upload=14738]: Next segment err: context deadline exceeded Aug 7 23:53:31.739205060 (14h13m31.328550047s) [upload=14738]: Done uploading segments... Aug 7 23:53:31.739217965 (14h13m31.328562952s) [upload=14738]: Upload finished. err: context deadline exceeded Duration: 6.594191246s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X..X.XXXX.X............................................XX] duration: 30.001404028s FAIL (2025-08-07 23:53:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 14h11m59.789927345s) [seg_pos={0 0}, upload=14721]: Done begin upload segment. Aug 7 23:52:00.200599942 (14h11m59.789944959s) [noise=true , address=77.90.31.19:4843 , piece_id=NVXWQQYUL3RSYHJZG53GE2JQMW7FU3HIUPGPJYYQAJGULOICBHEA, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.200609069 (14h11m59.789954056s) [noise=true , address=77.90.31.21:4879 , piece_id=CIOUHI4S3EYON4OJ5VMRK6E5QHBFVM3LDUUUPE7GNR27UJPXWXOQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.200700554 (14h11m59.790045551s) [noise=true , address=104.36.82.7:4910 , piece_id=H5P5BTNIZQVZ2OTW5TBWHQHC6PLJXNCLAS2RJNDJZIDIMNZT4FVQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.201562724 (14h11m59.790907721s) [noise=true , address=45.91.116.14:4191 , piece_id=CYG3IQ6A57K2BJJ7WBRN65RPEHKIHCRNV6FEC3NUGZNUEW3VMLPQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.202140499 (14h11m59.791485496s) [noise=true , address=77.90.31.18:4805 , piece_id=2Y55RTN4G7HSPAB734OGCQNHK7HFDSKGAYDBIQOGRGWB2JS7IWWQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.202565311 (14h11m59.791910298s) [noise=true , address=77.90.31.12:4590 , piece_id=FJBGEBSFJPFPVKZFJ6OVQQZGYQLE6TW3STY3ODGZQBEXUSYCUHHQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.202941321 (14h11m59.792286308s) [noise=true , address=45.91.116.8:4048 , piece_id=PFXTFJB64UCZPXY2MRJ3QNZYTBJPMGSCEINTZUEDAZ6BTLDMF4ZA, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.203062854 (14h11m59.792407861s) [noise=true , address=45.91.116.4:4938 , piece_id=POIEFCB5CBXZW3TKO3ZMT4ZEJJIWZHBUWOPXRLKBEGYB74IESQDQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.203542401 (14h11m59.792887378s) [noise=true , address=45.91.116.5:4297 , piece_id=CUMAL7XFD66GLBD4RAKP3SNBO4AIEYJXVTXSUNX3BQXUPQQ6S55A, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.203900947 (14h11m59.793245944s) [noise=true , address=77.90.31.14:4663 , piece_id=F6IYWOJJPLNENQ7KZ742UTPAAQIAPAWRSMLYULRZVJIS4NPW7UVA, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.204331041 (14h11m59.793676018s) [noise=true , address=77.90.31.19:4841 , piece_id=ZTUCRHKDBK4EGME3U2XXWMT36NISJKBJEYGNAUIIAAM53WBKVKZA, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.204730284 (14h11m59.794075271s) [noise=true , address=45.91.116.6:4975 , piece_id=XLM5WWXS4VTEPCK6NQB4EUOSFFRUVO4MLCQH3SF67G6FO2QJNARQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.204740444 (14h11m59.794085431s) [noise=true , address=45.91.116.12:4226 , piece_id=UVRNWUJCXD7Q2VK3A6T5QPNIC4PW2JD36W74VMGDWSEQV62NRB6A, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.205044235 (14h11m59.794389232s) [noise=true , address=45.91.116.11:4118 , piece_id=DUTNCBIML6HDKSKTHE2CPMU25RBJB2GPLHQU2TSTAKVSAXPUXXNQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.205322417 (14h11m59.794667394s) [noise=true , address=104.36.82.12:4150 , piece_id=D4TYX25GZDVSTSSKXAGVXVSGMZIKC5ZOSDEU6QTQ2CYMG5MT5IKQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.205847752 (14h11m59.795192749s) [noise=true , address=104.36.82.2:4732 , piece_id=EG6RUU2IRWRGSCPU25XV3ESK2LXY3TKO4MJPFUITSRWO3LBLAPLQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:02.068398504 (14h12m1.657743501s) [upload=14721]: Next returned nil segment Aug 7 23:52:02.068415778 (14h12m1.657760765s) [upload=14721]: Waiting for error group managing segments... Aug 7 23:52:02.367422114 (14h12m1.956767111s) [noise=true , address=104.36.82.12:4150 , piece_id=D4TYX25GZDVSTSSKXAGVXVSGMZIKC5ZOSDEU6QTQ2CYMG5MT5IKQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.422374943 (14h12m2.01171993s) [noise=true , address=77.90.31.19:4841 , piece_id=ZTUCRHKDBK4EGME3U2XXWMT36NISJKBJEYGNAUIIAAM53WBKVKZA, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.608759795 (14h12m2.198104782s) [noise=true , address=104.36.82.2:4732 , piece_id=EG6RUU2IRWRGSCPU25XV3ESK2LXY3TKO4MJPFUITSRWO3LBLAPLQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.639784323 (14h12m2.22912934s) [noise=true , address=77.90.31.19:4843 , piece_id=NVXWQQYUL3RSYHJZG53GE2JQMW7FU3HIUPGPJYYQAJGULOICBHEA, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.662434068 (14h12m2.251779055s) [noise=true , address=45.91.116.4:4938 , piece_id=POIEFCB5CBXZW3TKO3ZMT4ZEJJIWZHBUWOPXRLKBEGYB74IESQDQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.729318345 (14h12m2.318663333s) [noise=true , address=45.91.116.6:4975 , piece_id=XLM5WWXS4VTEPCK6NQB4EUOSFFRUVO4MLCQH3SF67G6FO2QJNARQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.774437961 (14h12m2.363782948s) [noise=true , address=45.91.116.8:4048 , piece_id=PFXTFJB64UCZPXY2MRJ3QNZYTBJPMGSCEINTZUEDAZ6BTLDMF4ZA, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.836024912 (14h12m2.425369899s) [noise=true , address=45.91.116.11:4118 , piece_id=DUTNCBIML6HDKSKTHE2CPMU25RBJB2GPLHQU2TSTAKVSAXPUXXNQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.847295246 (14h12m2.436640263s) [noise=true , address=104.36.82.7:4910 , piece_id=H5P5BTNIZQVZ2OTW5TBWHQHC6PLJXNCLAS2RJNDJZIDIMNZT4FVQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.848026284 (14h12m2.437371271s) [noise=true , address=77.90.31.12:4590 , piece_id=FJBGEBSFJPFPVKZFJ6OVQQZGYQLE6TW3STY3ODGZQBEXUSYCUHHQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.856271830 (14h12m2.445616818s) [noise=true , address=45.91.116.12:4226 , piece_id=UVRNWUJCXD7Q2VK3A6T5QPNIC4PW2JD36W74VMGDWSEQV62NRB6A, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.893735638 (14h12m2.483080626s) [noise=true , address=45.91.116.5:4297 , piece_id=CUMAL7XFD66GLBD4RAKP3SNBO4AIEYJXVTXSUNX3BQXUPQQ6S55A, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.938610556 (14h12m2.527955553s) [noise=true , address=77.90.31.14:4663 , piece_id=F6IYWOJJPLNENQ7KZ742UTPAAQIAPAWRSMLYULRZVJIS4NPW7UVA, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.938631105 (14h12m2.527976092s) [seg_pos={0 0}, upload=14721]: Segment reached optimal threshold of 13 pieces. Aug 7 23:52:02.938748910 (14h12m2.528093897s) [noise=true , address=77.90.31.21:4879 , piece_id=CIOUHI4S3EYON4OJ5VMRK6E5QHBFVM3LDUUUPE7GNR27UJPXWXOQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Aug 7 23:52:02.938875732 (14h12m2.528220729s) [noise=true , address=45.91.116.14:4191 , piece_id=CYG3IQ6A57K2BJJ7WBRN65RPEHKIHCRNV6FEC3NUGZNUEW3VMLPQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Aug 7 23:52:02.939590790 (14h12m2.528935778s) [noise=true , address=77.90.31.18:4805 , piece_id=2Y55RTN4G7HSPAB734OGCQNHK7HFDSKGAYDBIQOGRGWB2JS7IWWQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 7 23:52:02.939610088 (14h12m2.528955075s) [seg_pos={0 0}, upload=14721]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:52:02.939723344 (14h12m2.529068331s) [upload=14721]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:52:02.939730077 (14h12m2.529075064s) [upload=14721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:03.232136988 (14h12m2.821481985s) [upload=14721]: Done uploading segments... Aug 7 23:52:03.232187184 (14h12m2.821532171s) [upload=14721]: Upload finished. err: <nil> Aug 7 23:52:22.386769154 (14h12m21.976114141s) [upload=14726]: Starting upload Aug 7 23:52:22.386825732 (14h12m21.976170719s) [upload=14726]: Done starting upload Aug 7 23:52:22.386882942 (14h12m21.976227939s) [upload=14726]: Uploading segments... Aug 7 23:52:22.386993543 (14h12m21.97633853s) [upload=14726]: Got next segment. Inline: false Aug 7 23:52:22.387016918 (14h12m21.976361905s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:22.387020896 (14h12m21.976365883s) [upload=14726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:52:51.743802201 (14h12m51.333147178s) [upload=14726]: Next segment err: context deadline exceeded Aug 7 23:52:51.743938321 (14h12m51.333283298s) [upload=14726]: Done uploading segments... Aug 7 23:52:51.743958701 (14h12m51.333303688s) [upload=14726]: Upload finished. err: context deadline exceeded Aug 7 23:53:02.377428441 (14h13m1.966773458s) [upload=14738]: Starting upload Aug 7 23:53:02.377547469 (14h13m1.966892496s) [upload=14738]: Done starting upload Aug 7 23:53:02.377639475 (14h13m1.966984513s) [upload=14738]: Uploading segments... Aug 7 23:53:02.378016907 (14h13m1.967361894s) [upload=14738]: Got next segment. Inline: false Aug 7 23:53:02.378059268 (14h13m1.967404286s) [upload=14738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:02.378064859 (14h13m1.967409846s) [upload=14738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:53:31.738985270 (14h13m31.328330257s) [upload=14738]: Next segment err: context deadline exceeded Aug 7 23:53:31.739205060 (14h13m31.328550047s) [upload=14738]: Done uploading segments... Aug 7 23:53:31.739217965 (14h13m31.328562952s) [upload=14738]: Upload finished. err: context deadline exceeded Duration: 30.001426401s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.........XXXX.............................................XX] duration: 30.001081444s FAIL (2025-08-07 23:53:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad=14601]: Done uploading segments... Aug 7 23:45:24.942660848 (14h5m24.532005825s) [upload=14601]: Upload finished. err: <nil> Aug 7 23:46:21.408171737 (14h6m20.997516724s) [upload=14621]: Starting upload Aug 7 23:46:21.408226612 (14h6m20.997571599s) [upload=14621]: Done starting upload Aug 7 23:46:21.408277428 (14h6m20.997622415s) [upload=14621]: Uploading segments... Aug 7 23:46:21.408317405 (14h6m20.997662392s) [upload=14621]: Got next segment. Inline: true Aug 7 23:46:21.408321252 (14h6m20.99766624s) [upload=14621]: Waiting for error group managing segments... Aug 7 23:46:21.408355297 (14h6m20.997700284s) [upload=14621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:21.408359465 (14h6m20.997704452s) [upload=14621]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:21.408361911 (14h6m20.997706898s) [upload=14621]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:21.679074112 (14h6m21.268419099s) [upload=14621]: Done uploading segments... Aug 7 23:46:21.679092257 (14h6m21.268437234s) [upload=14621]: Upload finished. err: <nil> Aug 7 23:46:51.968618755 (14h6m51.557963762s) [upload=14630]: Starting upload Aug 7 23:46:51.968727564 (14h6m51.558072551s) [upload=14630]: Done starting upload Aug 7 23:46:51.968806966 (14h6m51.558151983s) [upload=14630]: Uploading segments... Aug 7 23:46:51.968857222 (14h6m51.558202209s) [upload=14630]: Got next segment. Inline: true Aug 7 23:46:51.968861520 (14h6m51.558206507s) [upload=14630]: Waiting for error group managing segments... Aug 7 23:46:51.968910524 (14h6m51.558255511s) [upload=14630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:51.968925081 (14h6m51.558270068s) [upload=14630]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:51.968929641 (14h6m51.558274638s) [upload=14630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:52.240213477 (14h6m51.829558464s) [upload=14630]: Done uploading segments... Aug 7 23:46:52.240231210 (14h6m51.829576197s) [upload=14630]: Upload finished. err: <nil> Aug 7 23:48:01.687516274 (14h8m1.276861261s) [upload=14647]: Starting upload Aug 7 23:48:01.687609412 (14h8m1.276954409s) [upload=14647]: Done starting upload Aug 7 23:48:01.687680398 (14h8m1.277025385s) [upload=14647]: Uploading segments... Aug 7 23:48:01.687725454 (14h8m1.277070441s) [upload=14647]: Got next segment. Inline: true Aug 7 23:48:01.687731175 (14h8m1.277076163s) [upload=14647]: Waiting for error group managing segments... Aug 7 23:48:01.687779207 (14h8m1.277124194s) [upload=14647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:01.687784667 (14h8m1.277129664s) [upload=14647]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:01.687788285 (14h8m1.277133282s) [upload=14647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:01.957323975 (14h8m1.546668972s) [upload=14647]: Done uploading segments... Aug 7 23:48:01.957343102 (14h8m1.546688089s) [upload=14647]: Upload finished. err: <nil> Aug 7 23:48:17.710466019 (14h8m17.299811006s) [upload=14652]: Starting upload Aug 7 23:48:17.710547815 (14h8m17.299892802s) [upload=14652]: Done starting upload Aug 7 23:48:17.710624521 (14h8m17.299969508s) [upload=14652]: Uploading segments... Aug 7 23:48:17.710729823 (14h8m17.30007481s) [upload=14652]: Got next segment. Inline: true Aug 7 23:48:17.710734742 (14h8m17.300079719s) [upload=14652]: Waiting for error group managing segments... Aug 7 23:48:17.710799587 (14h8m17.300144574s) [upload=14652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:17.710804295 (14h8m17.300149273s) [upload=14652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:17.710807071 (14h8m17.300152058s) [upload=14652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:17.976849249 (14h8m17.566194236s) [upload=14652]: Done uploading segments... Aug 7 23:48:17.976866912 (14h8m17.566211899s) [upload=14652]: Upload finished. err: <nil> Aug 7 23:49:05.340307173 (14h9m4.92965215s) [upload=14667]: Starting upload Aug 7 23:49:05.340383028 (14h9m4.929728015s) [upload=14667]: Done starting upload Aug 7 23:49:05.340707899 (14h9m4.930052887s) [upload=14667]: Uploading segments... Aug 7 23:49:05.340775519 (14h9m4.930120496s) [upload=14667]: Got next segment. Inline: true Aug 7 23:49:05.340781871 (14h9m4.930126858s) [upload=14667]: Waiting for error group managing segments... Aug 7 23:49:05.340825234 (14h9m4.930170221s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:05.340830374 (14h9m4.930175361s) [upload=14667]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:05.340833269 (14h9m4.930178246s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:05.817578876 (14h9m5.406923853s) [upload=14667]: Done uploading segments... Aug 7 23:49:05.817594757 (14h9m5.406939744s) [upload=14667]: Upload finished. err: <nil> Aug 7 23:49:28.341886774 (14h9m27.931231761s) [upload=14673]: Starting upload Aug 7 23:49:28.341998327 (14h9m27.931343334s) [upload=14673]: Done starting upload Aug 7 23:49:28.342069203 (14h9m27.9314142s) [upload=14673]: Uploading segments... Aug 7 23:49:28.342116133 (14h9m27.93146113s) [upload=14673]: Got next segment. Inline: true Aug 7 23:49:28.342121513 (14h9m27.93146651s) [upload=14673]: Waiting for error group managing segments... Aug 7 23:49:28.342171930 (14h9m27.931516927s) [upload=14673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:28.342178332 (14h9m27.931523319s) [upload=14673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:28.342182279 (14h9m27.931527256s) [upload=14673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:28.643582626 (14h9m28.232927603s) [upload=14673]: Done uploading segments... Aug 7 23:49:28.643602353 (14h9m28.23294734s) [upload=14673]: Upload finished. err: <nil> Aug 7 23:50:31.093627952 (14h10m30.682972939s) [upload=14692]: Starting upload Aug 7 23:50:31.093695632 (14h10m30.683040619s) [upload=14692]: Done starting upload Aug 7 23:50:31.093766667 (14h10m30.683111654s) [upload=14692]: Uploading segments... Aug 7 23:50:31.093857141 (14h10m30.683202128s) [upload=14692]: Got next segment. Inline: true Aug 7 23:50:31.093862081 (14h10m30.683207068s) [upload=14692]: Waiting for error group managing segments... Aug 7 23:50:31.093897037 (14h10m30.683242024s) [upload=14692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:31.093918809 (14h10m30.683263796s) [upload=14692]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:31.093921514 (14h10m30.683266501s) [upload=14692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:31.356040796 (14h10m30.945385783s) [upload=14692]: Done uploading segments... Aug 7 23:50:31.356106673 (14h10m30.94545166s) [upload=14692]: Upload finished. err: <nil> Aug 7 23:51:20.638658191 (14h11m20.228003178s) [upload=14707]: Starting upload Aug 7 23:51:20.638712213 (14h11m20.2280572s) [upload=14707]: Done starting upload Aug 7 23:51:20.638762700 (14h11m20.228107687s) [upload=14707]: Uploading segments... Aug 7 23:51:20.638799431 (14h11m20.228144418s) [upload=14707]: Got next segment. Inline: true Aug 7 23:51:20.638803058 (14h11m20.228148035s) [upload=14707]: Waiting for error group managing segments... Aug 7 23:51:20.638833967 (14h11m20.228178954s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:20.638838004 (14h11m20.228182991s) [upload=14707]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:20.638840399 (14h11m20.228185386s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:20.988587205 (14h11m20.577932192s) [upload=14707]: Done uploading segments... Aug 7 23:51:20.988605008 (14h11m20.577949985s) [upload=14707]: Upload finished. err: <nil> Aug 7 23:51:46.820025196 (14h11m46.409370183s) [upload=14715]: Starting upload Aug 7 23:51:46.820120348 (14h11m46.409465325s) [upload=14715]: Done starting upload Aug 7 23:51:46.820168981 (14h11m46.409513958s) [upload=14715]: Uploading segments... Aug 7 23:51:46.820463975 (14h11m46.409808972s) [upload=14715]: Got next segment. Inline: true Aug 7 23:51:46.820469005 (14h11m46.409813982s) [upload=14715]: Waiting for error group managing segments... Aug 7 23:51:46.820509402 (14h11m46.40985439s) [upload=14715]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:46.820513340 (14h11m46.409858327s) [upload=14715]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:46.820515594 (14h11m46.409860581s) [upload=14715]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:16.169845537 (14h12m15.759190524s) [upload=14715]: Done uploading segments... Aug 7 23:52:16.169877608 (14h12m15.759222595s) [upload=14715]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:52:54.182819305 (14h12m53.772164292s) [upload=14736]: Starting upload Aug 7 23:52:54.182879129 (14h12m53.772224116s) [upload=14736]: Done starting upload Aug 7 23:52:54.182920949 (14h12m53.772265926s) [upload=14736]: Uploading segments... Aug 7 23:52:54.182946839 (14h12m53.772291826s) [upload=14736]: Got next segment. Inline: true Aug 7 23:52:54.182950245 (14h12m53.772295222s) [upload=14736]: Waiting for error group managing segments... Aug 7 23:52:54.182983399 (14h12m53.772328386s) [upload=14736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:54.182987035 (14h12m53.772332022s) [upload=14736]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:54.182989680 (14h12m53.772334667s) [upload=14736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:23.541227271 (14h13m23.130591846s) [upload=14736]: Done uploading segments... Aug 7 23:53:23.541269933 (14h13m23.13061492s) [upload=14736]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001101502s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.....................X..............X........X.......] duration: 7.292458499s FAIL (2025-08-07 23:53:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 408ba8db7a0d8af05b7d5f8336d5c5f7f4f0063d72c98ffee3c421a5da299777 != before d0b3d19e255585f56d510f04d387098fcb83db6017871f75c2e11245b6143455 ======LOGS====== ished. err: <nil> Aug 7 23:48:24.457116068 (14h8m24.046461055s) [upload=14655]: Starting upload Aug 7 23:48:24.457179188 (14h8m24.046524175s) [upload=14655]: Done starting upload Aug 7 23:48:24.457293146 (14h8m24.046638133s) [upload=14655]: Uploading segments... Aug 7 23:48:24.457654307 (14h8m24.046999294s) [upload=14655]: Got next segment. Inline: false Aug 7 23:48:24.457709964 (14h8m24.047054951s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:24.457715013 (14h8m24.04706s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:48:24.901153105 (14h8m24.490498082s) [seg_pos={0 0}, upload=14655]: Begin upload segment... Aug 7 23:48:24.901189916 (14h8m24.490534903s) [seg_pos={0 0}, upload=14655]: Done begin upload segment. Aug 7 23:48:24.901208150 (14h8m24.490553137s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901547320 (14h8m24.490892297s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901866269 (14h8m24.491211257s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901979486 (14h8m24.491324473s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902167115 (14h8m24.491512102s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902393378 (14h8m24.491738355s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902443794 (14h8m24.491788771s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902663996 (14h8m24.492008983s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902931768 (14h8m24.492276755s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903278531 (14h8m24.492623518s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903535794 (14h8m24.492880781s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903745445 (14h8m24.493090432s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903975075 (14h8m24.493320062s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904281360 (14h8m24.493626337s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904481643 (14h8m24.49382663s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904668631 (14h8m24.494013608s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904852633 (14h8m24.49419762s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.905040943 (14h8m24.49438593s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:26.003752552 (14h8m25.593097539s) [upload=14655]: Next returned nil segment Aug 7 23:48:26.003765798 (14h8m25.593110785s) [upload=14655]: Waiting for error group managing segments... Aug 7 23:48:26.282577597 (14h8m25.871922574s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.322820552 (14h8m25.912165529s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.366928523 (14h8m25.95627351s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.822974095 (14h8m26.412319082s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.895356812 (14h8m26.4847018s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.915282265 (14h8m26.504627252s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.929180326 (14h8m26.518525313s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.126611753 (14h8m26.71595674s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.271660348 (14h8m26.861005335s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.274412033 (14h8m26.86375702s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.336714535 (14h8m26.926059522s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.759468572 (14h8m27.348813559s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.904353465 (14h8m27.493698452s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960895364 (14h8m27.550240341s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960922416 (14h8m27.550267403s) [seg_pos={0 0}, upload=14655]: Segment reached optimal threshold of 14 pieces. Aug 7 23:48:27.961026686 (14h8m27.550371673s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961266504 (14h8m27.550611491s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961284479 (14h8m27.550629456s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: rpc: dial tcp 104.36.82.5:4869: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:48:27.961415680 (14h8m27.550760667s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC): protocol: context canceled; context canceled Aug 7 23:48:27.961429706 (14h8m27.550774693s) [seg_pos={0 0}, upload=14655]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:48:27.961507495 (14h8m27.550852482s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:48:27.961511613 (14h8m27.5508566s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:28.311226257 (14h8m27.900571244s) [upload=14655]: Done uploading segments... Aug 7 23:48:28.311246106 (14h8m27.900591083s) [upload=14655]: Upload finished. err: <nil> Duration: 7.292482835s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........XXXX.X..XX.......................................X] duration: 30.001211446s FAIL (2025-08-07 23:53:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001233658s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [..............X.....X.X...................................X.] duration: 2.325875811s FAIL (2025-08-07 23:52:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.325897594s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X..X.XXXX.X............................................X] duration: 30.001281222s FAIL (2025-08-07 23:52:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 3:51:13.774589764 (14h11m13.363934771s) [upload=14703]: Done uploading segments... Aug 7 23:51:13.774614692 (14h11m13.363959679s) [upload=14703]: Upload finished. err: <nil> Aug 7 23:51:59.961837036 (14h11m59.551182023s) [upload=14721]: Starting upload Aug 7 23:51:59.961906209 (14h11m59.551251196s) [upload=14721]: Done starting upload Aug 7 23:51:59.962069772 (14h11m59.551414859s) [upload=14721]: Uploading segments... Aug 7 23:51:59.962200090 (14h11m59.551545077s) [upload=14721]: Got next segment. Inline: false Aug 7 23:51:59.962233856 (14h11m59.551578843s) [upload=14721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:59.962237983 (14h11m59.551582961s) [upload=14721]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:52:00.200529217 (14h11m59.789874254s) [seg_pos={0 0}, upload=14721]: Begin upload segment... Aug 7 23:52:00.200582358 (14h11m59.789927345s) [seg_pos={0 0}, upload=14721]: Done begin upload segment. Aug 7 23:52:00.200599942 (14h11m59.789944959s) [noise=true , address=77.90.31.19:4843 , piece_id=NVXWQQYUL3RSYHJZG53GE2JQMW7FU3HIUPGPJYYQAJGULOICBHEA, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.200609069 (14h11m59.789954056s) [noise=true , address=77.90.31.21:4879 , piece_id=CIOUHI4S3EYON4OJ5VMRK6E5QHBFVM3LDUUUPE7GNR27UJPXWXOQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.200700554 (14h11m59.790045551s) [noise=true , address=104.36.82.7:4910 , piece_id=H5P5BTNIZQVZ2OTW5TBWHQHC6PLJXNCLAS2RJNDJZIDIMNZT4FVQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.201562724 (14h11m59.790907721s) [noise=true , address=45.91.116.14:4191 , piece_id=CYG3IQ6A57K2BJJ7WBRN65RPEHKIHCRNV6FEC3NUGZNUEW3VMLPQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.202140499 (14h11m59.791485496s) [noise=true , address=77.90.31.18:4805 , piece_id=2Y55RTN4G7HSPAB734OGCQNHK7HFDSKGAYDBIQOGRGWB2JS7IWWQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.202565311 (14h11m59.791910298s) [noise=true , address=77.90.31.12:4590 , piece_id=FJBGEBSFJPFPVKZFJ6OVQQZGYQLE6TW3STY3ODGZQBEXUSYCUHHQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.202941321 (14h11m59.792286308s) [noise=true , address=45.91.116.8:4048 , piece_id=PFXTFJB64UCZPXY2MRJ3QNZYTBJPMGSCEINTZUEDAZ6BTLDMF4ZA, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.203062854 (14h11m59.792407861s) [noise=true , address=45.91.116.4:4938 , piece_id=POIEFCB5CBXZW3TKO3ZMT4ZEJJIWZHBUWOPXRLKBEGYB74IESQDQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.203542401 (14h11m59.792887378s) [noise=true , address=45.91.116.5:4297 , piece_id=CUMAL7XFD66GLBD4RAKP3SNBO4AIEYJXVTXSUNX3BQXUPQQ6S55A, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.203900947 (14h11m59.793245944s) [noise=true , address=77.90.31.14:4663 , piece_id=F6IYWOJJPLNENQ7KZ742UTPAAQIAPAWRSMLYULRZVJIS4NPW7UVA, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.204331041 (14h11m59.793676018s) [noise=true , address=77.90.31.19:4841 , piece_id=ZTUCRHKDBK4EGME3U2XXWMT36NISJKBJEYGNAUIIAAM53WBKVKZA, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.204730284 (14h11m59.794075271s) [noise=true , address=45.91.116.6:4975 , piece_id=XLM5WWXS4VTEPCK6NQB4EUOSFFRUVO4MLCQH3SF67G6FO2QJNARQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.204740444 (14h11m59.794085431s) [noise=true , address=45.91.116.12:4226 , piece_id=UVRNWUJCXD7Q2VK3A6T5QPNIC4PW2JD36W74VMGDWSEQV62NRB6A, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.205044235 (14h11m59.794389232s) [noise=true , address=45.91.116.11:4118 , piece_id=DUTNCBIML6HDKSKTHE2CPMU25RBJB2GPLHQU2TSTAKVSAXPUXXNQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.205322417 (14h11m59.794667394s) [noise=true , address=104.36.82.12:4150 , piece_id=D4TYX25GZDVSTSSKXAGVXVSGMZIKC5ZOSDEU6QTQ2CYMG5MT5IKQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:00.205847752 (14h11m59.795192749s) [noise=true , address=104.36.82.2:4732 , piece_id=EG6RUU2IRWRGSCPU25XV3ESK2LXY3TKO4MJPFUITSRWO3LBLAPLQ, seg_pos={0 0}, upload=14721]: Uploading piece... Aug 7 23:52:02.068398504 (14h12m1.657743501s) [upload=14721]: Next returned nil segment Aug 7 23:52:02.068415778 (14h12m1.657760765s) [upload=14721]: Waiting for error group managing segments... Aug 7 23:52:02.367422114 (14h12m1.956767111s) [noise=true , address=104.36.82.12:4150 , piece_id=D4TYX25GZDVSTSSKXAGVXVSGMZIKC5ZOSDEU6QTQ2CYMG5MT5IKQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.422374943 (14h12m2.01171993s) [noise=true , address=77.90.31.19:4841 , piece_id=ZTUCRHKDBK4EGME3U2XXWMT36NISJKBJEYGNAUIIAAM53WBKVKZA, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.608759795 (14h12m2.198104782s) [noise=true , address=104.36.82.2:4732 , piece_id=EG6RUU2IRWRGSCPU25XV3ESK2LXY3TKO4MJPFUITSRWO3LBLAPLQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.639784323 (14h12m2.22912934s) [noise=true , address=77.90.31.19:4843 , piece_id=NVXWQQYUL3RSYHJZG53GE2JQMW7FU3HIUPGPJYYQAJGULOICBHEA, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.662434068 (14h12m2.251779055s) [noise=true , address=45.91.116.4:4938 , piece_id=POIEFCB5CBXZW3TKO3ZMT4ZEJJIWZHBUWOPXRLKBEGYB74IESQDQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.729318345 (14h12m2.318663333s) [noise=true , address=45.91.116.6:4975 , piece_id=XLM5WWXS4VTEPCK6NQB4EUOSFFRUVO4MLCQH3SF67G6FO2QJNARQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.774437961 (14h12m2.363782948s) [noise=true , address=45.91.116.8:4048 , piece_id=PFXTFJB64UCZPXY2MRJ3QNZYTBJPMGSCEINTZUEDAZ6BTLDMF4ZA, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.836024912 (14h12m2.425369899s) [noise=true , address=45.91.116.11:4118 , piece_id=DUTNCBIML6HDKSKTHE2CPMU25RBJB2GPLHQU2TSTAKVSAXPUXXNQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.847295246 (14h12m2.436640263s) [noise=true , address=104.36.82.7:4910 , piece_id=H5P5BTNIZQVZ2OTW5TBWHQHC6PLJXNCLAS2RJNDJZIDIMNZT4FVQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.848026284 (14h12m2.437371271s) [noise=true , address=77.90.31.12:4590 , piece_id=FJBGEBSFJPFPVKZFJ6OVQQZGYQLE6TW3STY3ODGZQBEXUSYCUHHQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.856271830 (14h12m2.445616818s) [noise=true , address=45.91.116.12:4226 , piece_id=UVRNWUJCXD7Q2VK3A6T5QPNIC4PW2JD36W74VMGDWSEQV62NRB6A, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.893735638 (14h12m2.483080626s) [noise=true , address=45.91.116.5:4297 , piece_id=CUMAL7XFD66GLBD4RAKP3SNBO4AIEYJXVTXSUNX3BQXUPQQ6S55A, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.938610556 (14h12m2.527955553s) [noise=true , address=77.90.31.14:4663 , piece_id=F6IYWOJJPLNENQ7KZ742UTPAAQIAPAWRSMLYULRZVJIS4NPW7UVA, seg_pos={0 0}, upload=14721]: Done uploading piece. err: <nil> Aug 7 23:52:02.938631105 (14h12m2.527976092s) [seg_pos={0 0}, upload=14721]: Segment reached optimal threshold of 13 pieces. Aug 7 23:52:02.938748910 (14h12m2.528093897s) [noise=true , address=77.90.31.21:4879 , piece_id=CIOUHI4S3EYON4OJ5VMRK6E5QHBFVM3LDUUUPE7GNR27UJPXWXOQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Aug 7 23:52:02.938875732 (14h12m2.528220729s) [noise=true , address=45.91.116.14:4191 , piece_id=CYG3IQ6A57K2BJJ7WBRN65RPEHKIHCRNV6FEC3NUGZNUEW3VMLPQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Aug 7 23:52:02.939590790 (14h12m2.528935778s) [noise=true , address=77.90.31.18:4805 , piece_id=2Y55RTN4G7HSPAB734OGCQNHK7HFDSKGAYDBIQOGRGWB2JS7IWWQ, seg_pos={0 0}, upload=14721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 7 23:52:02.939610088 (14h12m2.528955075s) [seg_pos={0 0}, upload=14721]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:52:02.939723344 (14h12m2.529068331s) [upload=14721]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:52:02.939730077 (14h12m2.529075064s) [upload=14721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:03.232136988 (14h12m2.821481985s) [upload=14721]: Done uploading segments... Aug 7 23:52:03.232187184 (14h12m2.821532171s) [upload=14721]: Upload finished. err: <nil> Aug 7 23:52:22.386769154 (14h12m21.976114141s) [upload=14726]: Starting upload Aug 7 23:52:22.386825732 (14h12m21.976170719s) [upload=14726]: Done starting upload Aug 7 23:52:22.386882942 (14h12m21.976227939s) [upload=14726]: Uploading segments... Aug 7 23:52:22.386993543 (14h12m21.97633853s) [upload=14726]: Got next segment. Inline: false Aug 7 23:52:22.387016918 (14h12m21.976361905s) [upload=14726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:22.387020896 (14h12m21.976365883s) [upload=14726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:52:51.743802201 (14h12m51.333147178s) [upload=14726]: Next segment err: context deadline exceeded Aug 7 23:52:51.743938321 (14h12m51.333283298s) [upload=14726]: Done uploading segments... Aug 7 23:52:51.743958701 (14h12m51.333303688s) [upload=14726]: Upload finished. err: context deadline exceeded Duration: 30.001303223s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [......X...X..X.X............................................] duration: 1.993292365s FAIL (2025-08-07 23:52:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.993322283s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 648.076385ms target: https://us1.storj.io/login FAIL (2025-08-07 23:52:48 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 648.112755ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.....................X..............X........X......] duration: 8.072941356s FAIL (2025-08-07 23:52:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 408ba8db7a0d8af05b7d5f8336d5c5f7f4f0063d72c98ffee3c421a5da299777 != before d0b3d19e255585f56d510f04d387098fcb83db6017871f75c2e11245b6143455 ======LOGS====== ished. err: <nil> Aug 7 23:48:24.457116068 (14h8m24.046461055s) [upload=14655]: Starting upload Aug 7 23:48:24.457179188 (14h8m24.046524175s) [upload=14655]: Done starting upload Aug 7 23:48:24.457293146 (14h8m24.046638133s) [upload=14655]: Uploading segments... Aug 7 23:48:24.457654307 (14h8m24.046999294s) [upload=14655]: Got next segment. Inline: false Aug 7 23:48:24.457709964 (14h8m24.047054951s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:24.457715013 (14h8m24.04706s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:48:24.901153105 (14h8m24.490498082s) [seg_pos={0 0}, upload=14655]: Begin upload segment... Aug 7 23:48:24.901189916 (14h8m24.490534903s) [seg_pos={0 0}, upload=14655]: Done begin upload segment. Aug 7 23:48:24.901208150 (14h8m24.490553137s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901547320 (14h8m24.490892297s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901866269 (14h8m24.491211257s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901979486 (14h8m24.491324473s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902167115 (14h8m24.491512102s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902393378 (14h8m24.491738355s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902443794 (14h8m24.491788771s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902663996 (14h8m24.492008983s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902931768 (14h8m24.492276755s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903278531 (14h8m24.492623518s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903535794 (14h8m24.492880781s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903745445 (14h8m24.493090432s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903975075 (14h8m24.493320062s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904281360 (14h8m24.493626337s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904481643 (14h8m24.49382663s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904668631 (14h8m24.494013608s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904852633 (14h8m24.49419762s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.905040943 (14h8m24.49438593s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:26.003752552 (14h8m25.593097539s) [upload=14655]: Next returned nil segment Aug 7 23:48:26.003765798 (14h8m25.593110785s) [upload=14655]: Waiting for error group managing segments... Aug 7 23:48:26.282577597 (14h8m25.871922574s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.322820552 (14h8m25.912165529s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.366928523 (14h8m25.95627351s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.822974095 (14h8m26.412319082s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.895356812 (14h8m26.4847018s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.915282265 (14h8m26.504627252s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.929180326 (14h8m26.518525313s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.126611753 (14h8m26.71595674s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.271660348 (14h8m26.861005335s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.274412033 (14h8m26.86375702s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.336714535 (14h8m26.926059522s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.759468572 (14h8m27.348813559s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.904353465 (14h8m27.493698452s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960895364 (14h8m27.550240341s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960922416 (14h8m27.550267403s) [seg_pos={0 0}, upload=14655]: Segment reached optimal threshold of 14 pieces. Aug 7 23:48:27.961026686 (14h8m27.550371673s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961266504 (14h8m27.550611491s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961284479 (14h8m27.550629456s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: rpc: dial tcp 104.36.82.5:4869: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:48:27.961415680 (14h8m27.550760667s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC): protocol: context canceled; context canceled Aug 7 23:48:27.961429706 (14h8m27.550774693s) [seg_pos={0 0}, upload=14655]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:48:27.961507495 (14h8m27.550852482s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:48:27.961511613 (14h8m27.5508566s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:28.311226257 (14h8m27.900571244s) [upload=14655]: Done uploading segments... Aug 7 23:48:28.311246106 (14h8m27.900591083s) [upload=14655]: Upload finished. err: <nil> Duration: 8.072968048s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....X......X................................X..............] duration: 1.895015676s FAIL (2025-08-07 23:52:42 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14406]: Done uploading segments... Aug 7 23:33:49.820540362 (13h53m49.409885379s) [upload=14406]: Upload finished. err: <nil> Aug 7 23:34:55.875843674 (13h54m55.465188661s) [upload=14425]: Starting upload Aug 7 23:34:55.875888460 (13h54m55.465233447s) [upload=14425]: Done starting upload Aug 7 23:34:55.875931803 (13h54m55.46527678s) [upload=14425]: Uploading segments... Aug 7 23:34:55.875961150 (13h54m55.465306137s) [upload=14425]: Got next segment. Inline: true Aug 7 23:34:55.875964626 (13h54m55.465309603s) [upload=14425]: Waiting for error group managing segments... Aug 7 23:34:55.875996888 (13h54m55.465341875s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:34:55.876001015 (13h54m55.465346002s) [upload=14425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:34:55.876003680 (13h54m55.465348667s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:34:56.158049155 (13h54m55.747394142s) [upload=14425]: Done uploading segments... Aug 7 23:34:56.158066439 (13h54m55.747411426s) [upload=14425]: Upload finished. err: <nil> Aug 7 23:35:36.771198141 (13h55m36.360543128s) [upload=14436]: Starting upload Aug 7 23:35:36.771239510 (13h55m36.360584487s) [upload=14436]: Done starting upload Aug 7 23:35:36.771282783 (13h55m36.36062776s) [upload=14436]: Uploading segments... Aug 7 23:35:36.771311258 (13h55m36.360656245s) [upload=14436]: Got next segment. Inline: true Aug 7 23:35:36.771315045 (13h55m36.360660032s) [upload=14436]: Waiting for error group managing segments... Aug 7 23:35:36.771344952 (13h55m36.360689939s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:35:36.771350051 (13h55m36.360695038s) [upload=14436]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:35:36.771352185 (13h55m36.360697172s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:35:37.066875398 (13h55m36.656220385s) [upload=14436]: Done uploading segments... Aug 7 23:35:37.066894275 (13h55m36.656239262s) [upload=14436]: Upload finished. err: <nil> Aug 7 23:36:11.909385420 (13h56m11.498730407s) [upload=14444]: Starting upload Aug 7 23:36:11.909445175 (13h56m11.498790172s) [upload=14444]: Done starting upload Aug 7 23:36:11.909503978 (13h56m11.498848985s) [upload=14444]: Uploading segments... Aug 7 23:36:11.909562349 (13h56m11.498907356s) [upload=14444]: Got next segment. Inline: true Aug 7 23:36:11.909568220 (13h56m11.498913207s) [upload=14444]: Waiting for error group managing segments... Aug 7 23:36:11.909608778 (13h56m11.498953765s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:11.909614439 (13h56m11.498959416s) [upload=14444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:11.909617855 (13h56m11.498962842s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:12.226741776 (13h56m11.816086753s) [upload=14444]: Done uploading segments... Aug 7 23:36:12.226761765 (13h56m11.816106752s) [upload=14444]: Upload finished. err: <nil> Aug 7 23:36:45.061365955 (13h56m44.650710942s) [upload=14452]: Starting upload Aug 7 23:36:45.061414287 (13h56m44.650759274s) [upload=14452]: Done starting upload Aug 7 23:36:45.061459804 (13h56m44.650804792s) [upload=14452]: Uploading segments... Aug 7 23:36:45.061489772 (13h56m44.650834759s) [upload=14452]: Got next segment. Inline: true Aug 7 23:36:45.061493318 (13h56m44.650838305s) [upload=14452]: Waiting for error group managing segments... Aug 7 23:36:45.061525611 (13h56m44.650870598s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:45.061529408 (13h56m44.650874395s) [upload=14452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:45.061531542 (13h56m44.650876529s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:45.340854790 (13h56m44.930199777s) [upload=14452]: Done uploading segments... Aug 7 23:36:45.340871481 (13h56m44.930216468s) [upload=14452]: Upload finished. err: <nil> Aug 7 23:37:27.165030871 (13h57m26.754375858s) [upload=14466]: Starting upload Aug 7 23:37:27.165168204 (13h57m26.754513201s) [upload=14466]: Done starting upload Aug 7 23:37:27.165252034 (13h57m26.754597061s) [upload=14466]: Uploading segments... Aug 7 23:37:27.165304224 (13h57m26.754649251s) [upload=14466]: Got next segment. Inline: true Aug 7 23:37:27.165311979 (13h57m26.754657006s) [upload=14466]: Waiting for error group managing segments... Aug 7 23:37:27.165356895 (13h57m26.754701872s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:37:27.165361664 (13h57m26.754706651s) [upload=14466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:37:27.165364449 (13h57m26.754709426s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:37:27.453150839 (13h57m27.042495816s) [upload=14466]: Done uploading segments... Aug 7 23:37:27.453172210 (13h57m27.042517218s) [upload=14466]: Upload finished. err: <nil> Aug 7 23:38:33.292631280 (13h58m32.881976267s) [upload=14482]: Starting upload Aug 7 23:38:33.292682438 (13h58m32.882027425s) [upload=14482]: Done starting upload Aug 7 23:38:33.292735750 (13h58m32.882080737s) [upload=14482]: Uploading segments... Aug 7 23:38:33.292771078 (13h58m32.882116065s) [upload=14482]: Got next segment. Inline: true Aug 7 23:38:33.292774524 (13h58m32.882119501s) [upload=14482]: Waiting for error group managing segments... Aug 7 23:38:33.292809602 (13h58m32.882154589s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:33.292814000 (13h58m32.882158987s) [upload=14482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:33.292816806 (13h58m32.882161793s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:33.652660690 (13h58m33.242005677s) [upload=14482]: Done uploading segments... Aug 7 23:38:33.652677502 (13h58m33.242022489s) [upload=14482]: Upload finished. err: <nil> Aug 7 23:38:56.173816507 (13h58m55.763161484s) [upload=14490]: Starting upload Aug 7 23:38:56.173867385 (13h58m55.763212372s) [upload=14490]: Done starting upload Aug 7 23:38:56.173920446 (13h58m55.763265433s) [upload=14490]: Uploading segments... Aug 7 23:38:56.173947418 (13h58m55.763292405s) [upload=14490]: Got next segment. Inline: true Aug 7 23:38:56.173951646 (13h58m55.763296633s) [upload=14490]: Waiting for error group managing segments... Aug 7 23:38:56.173986452 (13h58m55.763331439s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:56.173990099 (13h58m55.763335086s) [upload=14490]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:56.173992515 (13h58m55.763337492s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:56.442927417 (13h58m56.032272424s) [upload=14490]: Done uploading segments... Aug 7 23:38:56.442950000 (13h58m56.032294987s) [upload=14490]: Upload finished. err: <nil> Aug 7 23:39:51.382592315 (13h59m50.971937292s) [upload=14506]: Starting upload Aug 7 23:39:51.382637612 (13h59m50.971982599s) [upload=14506]: Done starting upload Aug 7 23:39:51.382703347 (13h59m50.972048324s) [upload=14506]: Uploading segments... Aug 7 23:39:51.382762962 (13h59m50.972107949s) [upload=14506]: Got next segment. Inline: true Aug 7 23:39:51.382767260 (13h59m50.972112247s) [upload=14506]: Waiting for error group managing segments... Aug 7 23:39:51.382800283 (13h59m50.97214527s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:39:51.382803920 (13h59m50.972148907s) [upload=14506]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:39:51.382806144 (13h59m50.972151131s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:39:51.664212669 (13h59m51.253557746s) [upload=14506]: Done uploading segments... Aug 7 23:39:51.664238148 (13h59m51.253583125s) [upload=14506]: Upload finished. err: <nil> Aug 7 23:40:08.314581416 (14h0m7.903926403s) [upload=14509]: Starting upload Aug 7 23:40:08.314628847 (14h0m7.903973834s) [upload=14509]: Done starting upload Aug 7 23:40:08.314675916 (14h0m7.904020903s) [upload=14509]: Uploading segments... Aug 7 23:40:08.314709010 (14h0m7.904053998s) [upload=14509]: Got next segment. Inline: true Aug 7 23:40:08.314712467 (14h0m7.904057454s) [upload=14509]: Waiting for error group managing segments... Aug 7 23:40:08.314744288 (14h0m7.904089275s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:40:08.314748175 (14h0m7.904093162s) [upload=14509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:40:08.314750489 (14h0m7.904095476s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:40:08.646371868 (14h0m8.235716855s) [upload=14509]: Done uploading segments... Aug 7 23:40:08.646389852 (14h0m8.235734829s) [upload=14509]: Upload finished. err: <nil> Aug 7 23:41:18.965192405 (14h1m18.554537392s) [upload=14528]: Starting upload Aug 7 23:41:18.965250195 (14h1m18.554595172s) [upload=14528]: Done starting upload Aug 7 23:41:18.965313206 (14h1m18.554658193s) [upload=14528]: Uploading segments... Aug 7 23:41:18.965345408 (14h1m18.554690395s) [upload=14528]: Got next segment. Inline: true Aug 7 23:41:18.965349346 (14h1m18.554694333s) [upload=14528]: Waiting for error group managing segments... Aug 7 23:41:18.965382669 (14h1m18.554727656s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:41:18.965386546 (14h1m18.554731523s) [upload=14528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:41:18.965389191 (14h1m18.554734178s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:19.261666757 (14h1m18.851011744s) [upload=14528]: Done uploading segments... Aug 7 23:41:19.261683329 (14h1m18.851028316s) [upload=14528]: Upload finished. err: <nil> Duration: 1.895042918s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.879434141s FAIL (2025-08-07 23:52:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.879462777s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [..........XXXX.............................................X] duration: 30.00059396s FAIL (2025-08-07 23:52:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 44:32.061848431 (14h4m31.651193439s) [upload=14588]: Done uploading segments... Aug 7 23:44:32.061869903 (14h4m31.65121489s) [upload=14588]: Upload finished. err: <nil> Aug 7 23:45:24.675851582 (14h5m24.265196569s) [upload=14601]: Starting upload Aug 7 23:45:24.675899464 (14h5m24.265244451s) [upload=14601]: Done starting upload Aug 7 23:45:24.675950320 (14h5m24.265295307s) [upload=14601]: Uploading segments... Aug 7 23:45:24.675990638 (14h5m24.265335615s) [upload=14601]: Got next segment. Inline: true Aug 7 23:45:24.675994665 (14h5m24.265339652s) [upload=14601]: Waiting for error group managing segments... Aug 7 23:45:24.676023961 (14h5m24.265368938s) [upload=14601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:45:24.676028069 (14h5m24.265373056s) [upload=14601]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:45:24.676031756 (14h5m24.265376743s) [upload=14601]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:45:24.942643764 (14h5m24.531988752s) [upload=14601]: Done uploading segments... Aug 7 23:45:24.942660848 (14h5m24.532005825s) [upload=14601]: Upload finished. err: <nil> Aug 7 23:46:21.408171737 (14h6m20.997516724s) [upload=14621]: Starting upload Aug 7 23:46:21.408226612 (14h6m20.997571599s) [upload=14621]: Done starting upload Aug 7 23:46:21.408277428 (14h6m20.997622415s) [upload=14621]: Uploading segments... Aug 7 23:46:21.408317405 (14h6m20.997662392s) [upload=14621]: Got next segment. Inline: true Aug 7 23:46:21.408321252 (14h6m20.99766624s) [upload=14621]: Waiting for error group managing segments... Aug 7 23:46:21.408355297 (14h6m20.997700284s) [upload=14621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:21.408359465 (14h6m20.997704452s) [upload=14621]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:21.408361911 (14h6m20.997706898s) [upload=14621]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:21.679074112 (14h6m21.268419099s) [upload=14621]: Done uploading segments... Aug 7 23:46:21.679092257 (14h6m21.268437234s) [upload=14621]: Upload finished. err: <nil> Aug 7 23:46:51.968618755 (14h6m51.557963762s) [upload=14630]: Starting upload Aug 7 23:46:51.968727564 (14h6m51.558072551s) [upload=14630]: Done starting upload Aug 7 23:46:51.968806966 (14h6m51.558151983s) [upload=14630]: Uploading segments... Aug 7 23:46:51.968857222 (14h6m51.558202209s) [upload=14630]: Got next segment. Inline: true Aug 7 23:46:51.968861520 (14h6m51.558206507s) [upload=14630]: Waiting for error group managing segments... Aug 7 23:46:51.968910524 (14h6m51.558255511s) [upload=14630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:51.968925081 (14h6m51.558270068s) [upload=14630]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:51.968929641 (14h6m51.558274638s) [upload=14630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:52.240213477 (14h6m51.829558464s) [upload=14630]: Done uploading segments... Aug 7 23:46:52.240231210 (14h6m51.829576197s) [upload=14630]: Upload finished. err: <nil> Aug 7 23:48:01.687516274 (14h8m1.276861261s) [upload=14647]: Starting upload Aug 7 23:48:01.687609412 (14h8m1.276954409s) [upload=14647]: Done starting upload Aug 7 23:48:01.687680398 (14h8m1.277025385s) [upload=14647]: Uploading segments... Aug 7 23:48:01.687725454 (14h8m1.277070441s) [upload=14647]: Got next segment. Inline: true Aug 7 23:48:01.687731175 (14h8m1.277076163s) [upload=14647]: Waiting for error group managing segments... Aug 7 23:48:01.687779207 (14h8m1.277124194s) [upload=14647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:01.687784667 (14h8m1.277129664s) [upload=14647]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:01.687788285 (14h8m1.277133282s) [upload=14647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:01.957323975 (14h8m1.546668972s) [upload=14647]: Done uploading segments... Aug 7 23:48:01.957343102 (14h8m1.546688089s) [upload=14647]: Upload finished. err: <nil> Aug 7 23:48:17.710466019 (14h8m17.299811006s) [upload=14652]: Starting upload Aug 7 23:48:17.710547815 (14h8m17.299892802s) [upload=14652]: Done starting upload Aug 7 23:48:17.710624521 (14h8m17.299969508s) [upload=14652]: Uploading segments... Aug 7 23:48:17.710729823 (14h8m17.30007481s) [upload=14652]: Got next segment. Inline: true Aug 7 23:48:17.710734742 (14h8m17.300079719s) [upload=14652]: Waiting for error group managing segments... Aug 7 23:48:17.710799587 (14h8m17.300144574s) [upload=14652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:17.710804295 (14h8m17.300149273s) [upload=14652]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:17.710807071 (14h8m17.300152058s) [upload=14652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:17.976849249 (14h8m17.566194236s) [upload=14652]: Done uploading segments... Aug 7 23:48:17.976866912 (14h8m17.566211899s) [upload=14652]: Upload finished. err: <nil> Aug 7 23:49:05.340307173 (14h9m4.92965215s) [upload=14667]: Starting upload Aug 7 23:49:05.340383028 (14h9m4.929728015s) [upload=14667]: Done starting upload Aug 7 23:49:05.340707899 (14h9m4.930052887s) [upload=14667]: Uploading segments... Aug 7 23:49:05.340775519 (14h9m4.930120496s) [upload=14667]: Got next segment. Inline: true Aug 7 23:49:05.340781871 (14h9m4.930126858s) [upload=14667]: Waiting for error group managing segments... Aug 7 23:49:05.340825234 (14h9m4.930170221s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:05.340830374 (14h9m4.930175361s) [upload=14667]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:05.340833269 (14h9m4.930178246s) [upload=14667]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:05.817578876 (14h9m5.406923853s) [upload=14667]: Done uploading segments... Aug 7 23:49:05.817594757 (14h9m5.406939744s) [upload=14667]: Upload finished. err: <nil> Aug 7 23:49:28.341886774 (14h9m27.931231761s) [upload=14673]: Starting upload Aug 7 23:49:28.341998327 (14h9m27.931343334s) [upload=14673]: Done starting upload Aug 7 23:49:28.342069203 (14h9m27.9314142s) [upload=14673]: Uploading segments... Aug 7 23:49:28.342116133 (14h9m27.93146113s) [upload=14673]: Got next segment. Inline: true Aug 7 23:49:28.342121513 (14h9m27.93146651s) [upload=14673]: Waiting for error group managing segments... Aug 7 23:49:28.342171930 (14h9m27.931516927s) [upload=14673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:28.342178332 (14h9m27.931523319s) [upload=14673]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:28.342182279 (14h9m27.931527256s) [upload=14673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:28.643582626 (14h9m28.232927603s) [upload=14673]: Done uploading segments... Aug 7 23:49:28.643602353 (14h9m28.23294734s) [upload=14673]: Upload finished. err: <nil> Aug 7 23:50:31.093627952 (14h10m30.682972939s) [upload=14692]: Starting upload Aug 7 23:50:31.093695632 (14h10m30.683040619s) [upload=14692]: Done starting upload Aug 7 23:50:31.093766667 (14h10m30.683111654s) [upload=14692]: Uploading segments... Aug 7 23:50:31.093857141 (14h10m30.683202128s) [upload=14692]: Got next segment. Inline: true Aug 7 23:50:31.093862081 (14h10m30.683207068s) [upload=14692]: Waiting for error group managing segments... Aug 7 23:50:31.093897037 (14h10m30.683242024s) [upload=14692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:31.093918809 (14h10m30.683263796s) [upload=14692]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:31.093921514 (14h10m30.683266501s) [upload=14692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:31.356040796 (14h10m30.945385783s) [upload=14692]: Done uploading segments... Aug 7 23:50:31.356106673 (14h10m30.94545166s) [upload=14692]: Upload finished. err: <nil> Aug 7 23:51:20.638658191 (14h11m20.228003178s) [upload=14707]: Starting upload Aug 7 23:51:20.638712213 (14h11m20.2280572s) [upload=14707]: Done starting upload Aug 7 23:51:20.638762700 (14h11m20.228107687s) [upload=14707]: Uploading segments... Aug 7 23:51:20.638799431 (14h11m20.228144418s) [upload=14707]: Got next segment. Inline: true Aug 7 23:51:20.638803058 (14h11m20.228148035s) [upload=14707]: Waiting for error group managing segments... Aug 7 23:51:20.638833967 (14h11m20.228178954s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:20.638838004 (14h11m20.228182991s) [upload=14707]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:20.638840399 (14h11m20.228185386s) [upload=14707]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:20.988587205 (14h11m20.577932192s) [upload=14707]: Done uploading segments... Aug 7 23:51:20.988605008 (14h11m20.577949985s) [upload=14707]: Upload finished. err: <nil> Aug 7 23:51:46.820025196 (14h11m46.409370183s) [upload=14715]: Starting upload Aug 7 23:51:46.820120348 (14h11m46.409465325s) [upload=14715]: Done starting upload Aug 7 23:51:46.820168981 (14h11m46.409513958s) [upload=14715]: Uploading segments... Aug 7 23:51:46.820463975 (14h11m46.409808972s) [upload=14715]: Got next segment. Inline: true Aug 7 23:51:46.820469005 (14h11m46.409813982s) [upload=14715]: Waiting for error group managing segments... Aug 7 23:51:46.820509402 (14h11m46.40985439s) [upload=14715]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:46.820513340 (14h11m46.409858327s) [upload=14715]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:46.820515594 (14h11m46.409860581s) [upload=14715]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:16.169845537 (14h12m15.759190524s) [upload=14715]: Done uploading segments... Aug 7 23:52:16.169877608 (14h12m15.759222595s) [upload=14715]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000614629s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [...............X.....X.X...................................X] duration: 30.001084034s FAIL (2025-08-07 23:52:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001117448s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.023242653s FAIL (2025-08-07 23:52:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.0232716s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......X...X..X.X...........................................] duration: 1.858155864s FAIL (2025-08-07 23:52:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.858177025s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X..X.XXXX.X............................................] duration: 6.398336234s FAIL (2025-08-07 23:52:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 6.398361162s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.....................X..............X........X.....] duration: 7.273133027s FAIL (2025-08-07 23:52:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 408ba8db7a0d8af05b7d5f8336d5c5f7f4f0063d72c98ffee3c421a5da299777 != before d0b3d19e255585f56d510f04d387098fcb83db6017871f75c2e11245b6143455 ======LOGS====== ished. err: <nil> Aug 7 23:48:24.457116068 (14h8m24.046461055s) [upload=14655]: Starting upload Aug 7 23:48:24.457179188 (14h8m24.046524175s) [upload=14655]: Done starting upload Aug 7 23:48:24.457293146 (14h8m24.046638133s) [upload=14655]: Uploading segments... Aug 7 23:48:24.457654307 (14h8m24.046999294s) [upload=14655]: Got next segment. Inline: false Aug 7 23:48:24.457709964 (14h8m24.047054951s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:24.457715013 (14h8m24.04706s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:48:24.901153105 (14h8m24.490498082s) [seg_pos={0 0}, upload=14655]: Begin upload segment... Aug 7 23:48:24.901189916 (14h8m24.490534903s) [seg_pos={0 0}, upload=14655]: Done begin upload segment. Aug 7 23:48:24.901208150 (14h8m24.490553137s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901547320 (14h8m24.490892297s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901866269 (14h8m24.491211257s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901979486 (14h8m24.491324473s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902167115 (14h8m24.491512102s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902393378 (14h8m24.491738355s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902443794 (14h8m24.491788771s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902663996 (14h8m24.492008983s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902931768 (14h8m24.492276755s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903278531 (14h8m24.492623518s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903535794 (14h8m24.492880781s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903745445 (14h8m24.493090432s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903975075 (14h8m24.493320062s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904281360 (14h8m24.493626337s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904481643 (14h8m24.49382663s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904668631 (14h8m24.494013608s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904852633 (14h8m24.49419762s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.905040943 (14h8m24.49438593s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:26.003752552 (14h8m25.593097539s) [upload=14655]: Next returned nil segment Aug 7 23:48:26.003765798 (14h8m25.593110785s) [upload=14655]: Waiting for error group managing segments... Aug 7 23:48:26.282577597 (14h8m25.871922574s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.322820552 (14h8m25.912165529s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.366928523 (14h8m25.95627351s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.822974095 (14h8m26.412319082s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.895356812 (14h8m26.4847018s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.915282265 (14h8m26.504627252s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.929180326 (14h8m26.518525313s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.126611753 (14h8m26.71595674s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.271660348 (14h8m26.861005335s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.274412033 (14h8m26.86375702s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.336714535 (14h8m26.926059522s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.759468572 (14h8m27.348813559s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.904353465 (14h8m27.493698452s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960895364 (14h8m27.550240341s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960922416 (14h8m27.550267403s) [seg_pos={0 0}, upload=14655]: Segment reached optimal threshold of 14 pieces. Aug 7 23:48:27.961026686 (14h8m27.550371673s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961266504 (14h8m27.550611491s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961284479 (14h8m27.550629456s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: rpc: dial tcp 104.36.82.5:4869: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:48:27.961415680 (14h8m27.550760667s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC): protocol: context canceled; context canceled Aug 7 23:48:27.961429706 (14h8m27.550774693s) [seg_pos={0 0}, upload=14655]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:48:27.961507495 (14h8m27.550852482s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:48:27.961511613 (14h8m27.5508566s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:28.311226257 (14h8m27.900571244s) [upload=14655]: Done uploading segments... Aug 7 23:48:28.311246106 (14h8m27.900591083s) [upload=14655]: Upload finished. err: <nil> Duration: 7.273159198s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........XXXX.X..XX.......................................] duration: 6.587065373s FAIL (2025-08-07 23:52:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.58708973s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [.........XXXXX..............................................] duration: 1.996672029s FAIL (2025-08-07 23:51:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.996695696s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.........XXXXXX.X.X.........................................] duration: 5.749378993s FAIL (2025-08-07 23:51:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.749400324s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)