Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.327093292s FAIL (2025-08-10 22:55:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 6.327125905s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X..X...............X.......X.X.....X.X...........XX....] duration: 6.659519462s FAIL (2025-08-10 22:55:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 10 22:52:30.226093181 (53h26m24.657173312s) [upload=55724]: Starting upload Aug 10 22:52:30.226146663 (53h26m24.657226794s) [upload=55724]: Done starting upload Aug 10 22:52:30.226228670 (53h26m24.657308801s) [upload=55724]: Uploading segments... Aug 10 22:52:30.226383958 (53h26m24.657464089s) [upload=55724]: Got next segment. Inline: false Aug 10 22:52:30.226415327 (53h26m24.657495458s) [upload=55724]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:52:30.226421208 (53h26m24.65750134s) [upload=55724]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:52:30.677132253 (53h26m25.108212384s) [seg_pos={0 0}, upload=55724]: Begin upload segment... Aug 10 22:52:30.677190715 (53h26m25.108270846s) [seg_pos={0 0}, upload=55724]: Done begin upload segment. Aug 10 22:52:30.677225091 (53h26m25.108305212s) [noise=true , address=104.36.82.46:4507 , piece_id=B4NHEAVESRXDYUIOJ7A4WKZJOK4NPFT2JADD3JZXBPDVULAUEPIA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.677252474 (53h26m25.108332605s) [noise=true , address=45.91.116.14:4221 , piece_id=GAKJS3G242VOGC66CVIKCPTX7ITXYM2SQFVJPUAVJHJI36DIZ4BQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.677613114 (53h26m25.108693245s) [noise=true , address=104.36.82.45:4913 , piece_id=QA35NTBOIGIZWNNTBFZXECHK4SQUHBVAHWPCXNAH2YAAERZEJHMQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678313253 (53h26m25.109393384s) [noise=true , address=64.49.13.169:4713 , piece_id=7NZUZRKMLSKQ2YCASM264IOLWXRM5ZLQYAP53HL5BD52BLBCAEPA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678331708 (53h26m25.109411839s) [noise=true , address=104.36.82.35:4583 , piece_id=IXCJX5XZWILRXQMF24MUABV3TV2TW23R3T73XO563FKSTCDCYIYQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678613808 (53h26m25.109693939s) [noise=true , address=104.36.82.34:4564 , piece_id=AVAXGPPYXYSSZIIG47NV4RISUD4RPP7S6KTYF4QYSAPDM3NYOBDA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678652442 (53h26m25.109732573s) [noise=true , address=104.36.82.47:4549 , piece_id=IOXOLG2NU2H5CZIEXKPLH4EIJVKK4GX66SZQPDSCNPPKSOGFTUYA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678866331 (53h26m25.109946462s) [noise=true , address=104.36.82.20:4453 , piece_id=LHTZE3WPC3N5RCNUZKGMPKW3EYUR2FPVB2G6DFL6PL4ZECRYBAIA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679169000 (53h26m25.110249131s) [noise=true , address=104.36.82.61:4378 , piece_id=KHNTUED6WVXAB6A7NZOE3JIUMY6FPQOJECGFWQG6ATWVCKMW2NVQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679440199 (53h26m25.11052033s) [noise=true , address=104.36.82.36:4638 , piece_id=XUABMR23MOZG5QXDR2P3R5BSO5HHRZOP5QVSLKIZADSOZGZZIZNQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679713732 (53h26m25.110793863s) [noise=true , address=104.36.82.37:4663 , piece_id=3IGY7SFDVZYNVVGICK3C2XRBA5LRW5OQVZDFKODY5LKW57HWUR7A, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679926259 (53h26m25.11100639s) [noise=true , address=104.36.82.60:4348 , piece_id=Z4WXOGSIHUUIZPFDCJYA7ACUQXPXZWCT5QH7QLXR65T6AP2HIBIQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680126402 (53h26m25.111206533s) [noise=true , address=104.36.82.56:4238 , piece_id=7YLBQEL63SEYD2OILQ2PRKEP7U46PT4PSPAASSV5DJPAYTESOJ5A, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680326895 (53h26m25.111407026s) [noise=true , address=104.36.82.8:4995 , piece_id=G3K3LCGGAPEURR4G7HCBRM36ZBI4Q7MY2X5YT4CJLS2LRDRKXE7Q, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680530465 (53h26m25.111610596s) [noise=true , address=104.36.82.9:4030 , piece_id=7NIT67HJX2R7FUFLSSW36R54PJFKCGAKM5YJL6QWTO46UAY7GI5A, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680733313 (53h26m25.111813444s) [noise=true , address=104.36.82.16:4310 , piece_id=OXZLVTO5EASBCCHBSGFSHQSF2AI2U2XI6ENEIZPWN6T5YM7I2SLQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.681408626 (53h26m25.112488777s) [noise=true , address=104.36.82.44:4888 , piece_id=DPVGIMFES6RWUHO5OKRYJVX2HEDBBQJZHYYELVLMCBOWUKP6IKJA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.681726623 (53h26m25.112806755s) [noise=true , address=104.36.82.39:4746 , piece_id=UY4KKODNGNKAFOULMRSDCYJDT5HW7C6KYUD4EKZY3SBLXZRALOTQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:31.876216927 (53h26m26.307297058s) [upload=55724]: Next returned nil segment Aug 10 22:52:31.876232165 (53h26m26.307312296s) [upload=55724]: Waiting for error group managing segments... Aug 10 22:52:32.182969622 (53h26m26.614049743s) [noise=true , address=64.49.13.169:4713 , piece_id=7NZUZRKMLSKQ2YCASM264IOLWXRM5ZLQYAP53HL5BD52BLBCAEPA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.631903297 (53h26m27.062983428s) [noise=true , address=104.36.82.46:4507 , piece_id=B4NHEAVESRXDYUIOJ7A4WKZJOK4NPFT2JADD3JZXBPDVULAUEPIA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.652429340 (53h26m27.083509471s) [noise=true , address=104.36.82.34:4564 , piece_id=AVAXGPPYXYSSZIIG47NV4RISUD4RPP7S6KTYF4QYSAPDM3NYOBDA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.695199823 (53h26m27.126279954s) [noise=true , address=104.36.82.16:4310 , piece_id=OXZLVTO5EASBCCHBSGFSHQSF2AI2U2XI6ENEIZPWN6T5YM7I2SLQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.740711713 (53h26m27.171791844s) [noise=true , address=104.36.82.36:4638 , piece_id=XUABMR23MOZG5QXDR2P3R5BSO5HHRZOP5QVSLKIZADSOZGZZIZNQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.776799260 (53h26m27.207879391s) [noise=true , address=45.91.116.14:4221 , piece_id=GAKJS3G242VOGC66CVIKCPTX7ITXYM2SQFVJPUAVJHJI36DIZ4BQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.823959072 (53h26m27.255039203s) [noise=true , address=104.36.82.44:4888 , piece_id=DPVGIMFES6RWUHO5OKRYJVX2HEDBBQJZHYYELVLMCBOWUKP6IKJA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.873921155 (53h26m27.305001286s) [noise=true , address=104.36.82.35:4583 , piece_id=IXCJX5XZWILRXQMF24MUABV3TV2TW23R3T73XO563FKSTCDCYIYQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.886211631 (53h26m27.317291762s) [noise=true , address=104.36.82.45:4913 , piece_id=QA35NTBOIGIZWNNTBFZXECHK4SQUHBVAHWPCXNAH2YAAERZEJHMQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.886240236 (53h26m27.317320367s) [noise=true , address=104.36.82.47:4549 , piece_id=IOXOLG2NU2H5CZIEXKPLH4EIJVKK4GX66SZQPDSCNPPKSOGFTUYA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:33.361231516 (53h26m27.792311647s) [noise=true , address=104.36.82.9:4030 , piece_id=7NIT67HJX2R7FUFLSSW36R54PJFKCGAKM5YJL6QWTO46UAY7GI5A, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.095423782 (53h26m28.526503913s) [noise=true , address=104.36.82.20:4453 , piece_id=LHTZE3WPC3N5RCNUZKGMPKW3EYUR2FPVB2G6DFL6PL4ZECRYBAIA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.191766598 (53h26m28.622846729s) [noise=true , address=104.36.82.61:4378 , piece_id=KHNTUED6WVXAB6A7NZOE3JIUMY6FPQOJECGFWQG6ATWVCKMW2NVQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.957234535 (53h26m29.388314666s) [noise=true , address=104.36.82.39:4746 , piece_id=UY4KKODNGNKAFOULMRSDCYJDT5HW7C6KYUD4EKZY3SBLXZRALOTQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.957270263 (53h26m29.388350384s) [seg_pos={0 0}, upload=55724]: Segment reached optimal threshold of 14 pieces. Aug 10 22:52:34.957519320 (53h26m29.388599441s) [noise=true , address=104.36.82.60:4348 , piece_id=Z4WXOGSIHUUIZPFDCJYA7ACUQXPXZWCT5QH7QLXR65T6AP2HIBIQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: context canceled; context canceled Aug 10 22:52:34.957638617 (53h26m29.388718748s) [noise=true , address=104.36.82.37:4663 , piece_id=3IGY7SFDVZYNVVGICK3C2XRBA5LRW5OQVZDFKODY5LKW57HWUR7A, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: read tcp 188.166.225.170:38236->104.36.82.37:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:52:34.957696528 (53h26m29.388776659s) [noise=true , address=104.36.82.56:4238 , piece_id=7YLBQEL63SEYD2OILQ2PRKEP7U46PT4PSPAASSV5DJPAYTESOJ5A, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5): protocol: write tcp 188.166.225.170:55512->104.36.82.56:4238: use of closed network connection; write tcp 188.166.225.170:55512->104.36.82.56:4238: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55512->104.36.82.56:4238: use of closed network connection Aug 10 22:52:34.957729272 (53h26m29.388809393s) [noise=true , address=104.36.82.8:4995 , piece_id=G3K3LCGGAPEURR4G7HCBRM36ZBI4Q7MY2X5YT4CJLS2LRDRKXE7Q, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: rpc: dial tcp 104.36.82.8:4995: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:52:34.957748227 (53h26m29.388828348s) [seg_pos={0 0}, upload=55724]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:52:34.957846646 (53h26m29.388926767s) [upload=55724]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:52:34.957851144 (53h26m29.388931275s) [upload=55724]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:52:35.306840333 (53h26m29.737920484s) [upload=55724]: Done uploading segments... Aug 10 22:52:35.306857145 (53h26m29.737937276s) [upload=55724]: Upload finished. err: <nil> Duration: 6.659540391s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 8.812218089s FAIL (2025-08-10 22:55:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 8.812250551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.879998224s FAIL (2025-08-10 22:54:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.880030124s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 7.537986749s FAIL (2025-08-10 22:54:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 7.538004364s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X..X...............X.......X.X.....X.X...........XX...] duration: 6.578284153s FAIL (2025-08-10 22:54:24 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 10 22:52:30.226093181 (53h26m24.657173312s) [upload=55724]: Starting upload Aug 10 22:52:30.226146663 (53h26m24.657226794s) [upload=55724]: Done starting upload Aug 10 22:52:30.226228670 (53h26m24.657308801s) [upload=55724]: Uploading segments... Aug 10 22:52:30.226383958 (53h26m24.657464089s) [upload=55724]: Got next segment. Inline: false Aug 10 22:52:30.226415327 (53h26m24.657495458s) [upload=55724]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:52:30.226421208 (53h26m24.65750134s) [upload=55724]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:52:30.677132253 (53h26m25.108212384s) [seg_pos={0 0}, upload=55724]: Begin upload segment... Aug 10 22:52:30.677190715 (53h26m25.108270846s) [seg_pos={0 0}, upload=55724]: Done begin upload segment. Aug 10 22:52:30.677225091 (53h26m25.108305212s) [noise=true , address=104.36.82.46:4507 , piece_id=B4NHEAVESRXDYUIOJ7A4WKZJOK4NPFT2JADD3JZXBPDVULAUEPIA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.677252474 (53h26m25.108332605s) [noise=true , address=45.91.116.14:4221 , piece_id=GAKJS3G242VOGC66CVIKCPTX7ITXYM2SQFVJPUAVJHJI36DIZ4BQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.677613114 (53h26m25.108693245s) [noise=true , address=104.36.82.45:4913 , piece_id=QA35NTBOIGIZWNNTBFZXECHK4SQUHBVAHWPCXNAH2YAAERZEJHMQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678313253 (53h26m25.109393384s) [noise=true , address=64.49.13.169:4713 , piece_id=7NZUZRKMLSKQ2YCASM264IOLWXRM5ZLQYAP53HL5BD52BLBCAEPA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678331708 (53h26m25.109411839s) [noise=true , address=104.36.82.35:4583 , piece_id=IXCJX5XZWILRXQMF24MUABV3TV2TW23R3T73XO563FKSTCDCYIYQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678613808 (53h26m25.109693939s) [noise=true , address=104.36.82.34:4564 , piece_id=AVAXGPPYXYSSZIIG47NV4RISUD4RPP7S6KTYF4QYSAPDM3NYOBDA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678652442 (53h26m25.109732573s) [noise=true , address=104.36.82.47:4549 , piece_id=IOXOLG2NU2H5CZIEXKPLH4EIJVKK4GX66SZQPDSCNPPKSOGFTUYA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678866331 (53h26m25.109946462s) [noise=true , address=104.36.82.20:4453 , piece_id=LHTZE3WPC3N5RCNUZKGMPKW3EYUR2FPVB2G6DFL6PL4ZECRYBAIA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679169000 (53h26m25.110249131s) [noise=true , address=104.36.82.61:4378 , piece_id=KHNTUED6WVXAB6A7NZOE3JIUMY6FPQOJECGFWQG6ATWVCKMW2NVQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679440199 (53h26m25.11052033s) [noise=true , address=104.36.82.36:4638 , piece_id=XUABMR23MOZG5QXDR2P3R5BSO5HHRZOP5QVSLKIZADSOZGZZIZNQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679713732 (53h26m25.110793863s) [noise=true , address=104.36.82.37:4663 , piece_id=3IGY7SFDVZYNVVGICK3C2XRBA5LRW5OQVZDFKODY5LKW57HWUR7A, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679926259 (53h26m25.11100639s) [noise=true , address=104.36.82.60:4348 , piece_id=Z4WXOGSIHUUIZPFDCJYA7ACUQXPXZWCT5QH7QLXR65T6AP2HIBIQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680126402 (53h26m25.111206533s) [noise=true , address=104.36.82.56:4238 , piece_id=7YLBQEL63SEYD2OILQ2PRKEP7U46PT4PSPAASSV5DJPAYTESOJ5A, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680326895 (53h26m25.111407026s) [noise=true , address=104.36.82.8:4995 , piece_id=G3K3LCGGAPEURR4G7HCBRM36ZBI4Q7MY2X5YT4CJLS2LRDRKXE7Q, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680530465 (53h26m25.111610596s) [noise=true , address=104.36.82.9:4030 , piece_id=7NIT67HJX2R7FUFLSSW36R54PJFKCGAKM5YJL6QWTO46UAY7GI5A, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680733313 (53h26m25.111813444s) [noise=true , address=104.36.82.16:4310 , piece_id=OXZLVTO5EASBCCHBSGFSHQSF2AI2U2XI6ENEIZPWN6T5YM7I2SLQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.681408626 (53h26m25.112488777s) [noise=true , address=104.36.82.44:4888 , piece_id=DPVGIMFES6RWUHO5OKRYJVX2HEDBBQJZHYYELVLMCBOWUKP6IKJA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.681726623 (53h26m25.112806755s) [noise=true , address=104.36.82.39:4746 , piece_id=UY4KKODNGNKAFOULMRSDCYJDT5HW7C6KYUD4EKZY3SBLXZRALOTQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:31.876216927 (53h26m26.307297058s) [upload=55724]: Next returned nil segment Aug 10 22:52:31.876232165 (53h26m26.307312296s) [upload=55724]: Waiting for error group managing segments... Aug 10 22:52:32.182969622 (53h26m26.614049743s) [noise=true , address=64.49.13.169:4713 , piece_id=7NZUZRKMLSKQ2YCASM264IOLWXRM5ZLQYAP53HL5BD52BLBCAEPA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.631903297 (53h26m27.062983428s) [noise=true , address=104.36.82.46:4507 , piece_id=B4NHEAVESRXDYUIOJ7A4WKZJOK4NPFT2JADD3JZXBPDVULAUEPIA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.652429340 (53h26m27.083509471s) [noise=true , address=104.36.82.34:4564 , piece_id=AVAXGPPYXYSSZIIG47NV4RISUD4RPP7S6KTYF4QYSAPDM3NYOBDA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.695199823 (53h26m27.126279954s) [noise=true , address=104.36.82.16:4310 , piece_id=OXZLVTO5EASBCCHBSGFSHQSF2AI2U2XI6ENEIZPWN6T5YM7I2SLQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.740711713 (53h26m27.171791844s) [noise=true , address=104.36.82.36:4638 , piece_id=XUABMR23MOZG5QXDR2P3R5BSO5HHRZOP5QVSLKIZADSOZGZZIZNQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.776799260 (53h26m27.207879391s) [noise=true , address=45.91.116.14:4221 , piece_id=GAKJS3G242VOGC66CVIKCPTX7ITXYM2SQFVJPUAVJHJI36DIZ4BQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.823959072 (53h26m27.255039203s) [noise=true , address=104.36.82.44:4888 , piece_id=DPVGIMFES6RWUHO5OKRYJVX2HEDBBQJZHYYELVLMCBOWUKP6IKJA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.873921155 (53h26m27.305001286s) [noise=true , address=104.36.82.35:4583 , piece_id=IXCJX5XZWILRXQMF24MUABV3TV2TW23R3T73XO563FKSTCDCYIYQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.886211631 (53h26m27.317291762s) [noise=true , address=104.36.82.45:4913 , piece_id=QA35NTBOIGIZWNNTBFZXECHK4SQUHBVAHWPCXNAH2YAAERZEJHMQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.886240236 (53h26m27.317320367s) [noise=true , address=104.36.82.47:4549 , piece_id=IOXOLG2NU2H5CZIEXKPLH4EIJVKK4GX66SZQPDSCNPPKSOGFTUYA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:33.361231516 (53h26m27.792311647s) [noise=true , address=104.36.82.9:4030 , piece_id=7NIT67HJX2R7FUFLSSW36R54PJFKCGAKM5YJL6QWTO46UAY7GI5A, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.095423782 (53h26m28.526503913s) [noise=true , address=104.36.82.20:4453 , piece_id=LHTZE3WPC3N5RCNUZKGMPKW3EYUR2FPVB2G6DFL6PL4ZECRYBAIA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.191766598 (53h26m28.622846729s) [noise=true , address=104.36.82.61:4378 , piece_id=KHNTUED6WVXAB6A7NZOE3JIUMY6FPQOJECGFWQG6ATWVCKMW2NVQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.957234535 (53h26m29.388314666s) [noise=true , address=104.36.82.39:4746 , piece_id=UY4KKODNGNKAFOULMRSDCYJDT5HW7C6KYUD4EKZY3SBLXZRALOTQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.957270263 (53h26m29.388350384s) [seg_pos={0 0}, upload=55724]: Segment reached optimal threshold of 14 pieces. Aug 10 22:52:34.957519320 (53h26m29.388599441s) [noise=true , address=104.36.82.60:4348 , piece_id=Z4WXOGSIHUUIZPFDCJYA7ACUQXPXZWCT5QH7QLXR65T6AP2HIBIQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: context canceled; context canceled Aug 10 22:52:34.957638617 (53h26m29.388718748s) [noise=true , address=104.36.82.37:4663 , piece_id=3IGY7SFDVZYNVVGICK3C2XRBA5LRW5OQVZDFKODY5LKW57HWUR7A, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: read tcp 188.166.225.170:38236->104.36.82.37:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:52:34.957696528 (53h26m29.388776659s) [noise=true , address=104.36.82.56:4238 , piece_id=7YLBQEL63SEYD2OILQ2PRKEP7U46PT4PSPAASSV5DJPAYTESOJ5A, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5): protocol: write tcp 188.166.225.170:55512->104.36.82.56:4238: use of closed network connection; write tcp 188.166.225.170:55512->104.36.82.56:4238: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55512->104.36.82.56:4238: use of closed network connection Aug 10 22:52:34.957729272 (53h26m29.388809393s) [noise=true , address=104.36.82.8:4995 , piece_id=G3K3LCGGAPEURR4G7HCBRM36ZBI4Q7MY2X5YT4CJLS2LRDRKXE7Q, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: rpc: dial tcp 104.36.82.8:4995: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:52:34.957748227 (53h26m29.388828348s) [seg_pos={0 0}, upload=55724]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:52:34.957846646 (53h26m29.388926767s) [upload=55724]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:52:34.957851144 (53h26m29.388931275s) [upload=55724]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:52:35.306840333 (53h26m29.737920484s) [upload=55724]: Done uploading segments... Aug 10 22:52:35.306857145 (53h26m29.737937276s) [upload=55724]: Upload finished. err: <nil> Duration: 6.578308979s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.843693109s FAIL (2025-08-10 22:54:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.843717645s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......X..X...............X.......X.X.....X.X...........XX..] duration: 7.680845223s FAIL (2025-08-10 22:54:04 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 10 22:52:30.226093181 (53h26m24.657173312s) [upload=55724]: Starting upload Aug 10 22:52:30.226146663 (53h26m24.657226794s) [upload=55724]: Done starting upload Aug 10 22:52:30.226228670 (53h26m24.657308801s) [upload=55724]: Uploading segments... Aug 10 22:52:30.226383958 (53h26m24.657464089s) [upload=55724]: Got next segment. Inline: false Aug 10 22:52:30.226415327 (53h26m24.657495458s) [upload=55724]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:52:30.226421208 (53h26m24.65750134s) [upload=55724]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:52:30.677132253 (53h26m25.108212384s) [seg_pos={0 0}, upload=55724]: Begin upload segment... Aug 10 22:52:30.677190715 (53h26m25.108270846s) [seg_pos={0 0}, upload=55724]: Done begin upload segment. Aug 10 22:52:30.677225091 (53h26m25.108305212s) [noise=true , address=104.36.82.46:4507 , piece_id=B4NHEAVESRXDYUIOJ7A4WKZJOK4NPFT2JADD3JZXBPDVULAUEPIA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.677252474 (53h26m25.108332605s) [noise=true , address=45.91.116.14:4221 , piece_id=GAKJS3G242VOGC66CVIKCPTX7ITXYM2SQFVJPUAVJHJI36DIZ4BQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.677613114 (53h26m25.108693245s) [noise=true , address=104.36.82.45:4913 , piece_id=QA35NTBOIGIZWNNTBFZXECHK4SQUHBVAHWPCXNAH2YAAERZEJHMQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678313253 (53h26m25.109393384s) [noise=true , address=64.49.13.169:4713 , piece_id=7NZUZRKMLSKQ2YCASM264IOLWXRM5ZLQYAP53HL5BD52BLBCAEPA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678331708 (53h26m25.109411839s) [noise=true , address=104.36.82.35:4583 , piece_id=IXCJX5XZWILRXQMF24MUABV3TV2TW23R3T73XO563FKSTCDCYIYQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678613808 (53h26m25.109693939s) [noise=true , address=104.36.82.34:4564 , piece_id=AVAXGPPYXYSSZIIG47NV4RISUD4RPP7S6KTYF4QYSAPDM3NYOBDA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678652442 (53h26m25.109732573s) [noise=true , address=104.36.82.47:4549 , piece_id=IOXOLG2NU2H5CZIEXKPLH4EIJVKK4GX66SZQPDSCNPPKSOGFTUYA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678866331 (53h26m25.109946462s) [noise=true , address=104.36.82.20:4453 , piece_id=LHTZE3WPC3N5RCNUZKGMPKW3EYUR2FPVB2G6DFL6PL4ZECRYBAIA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679169000 (53h26m25.110249131s) [noise=true , address=104.36.82.61:4378 , piece_id=KHNTUED6WVXAB6A7NZOE3JIUMY6FPQOJECGFWQG6ATWVCKMW2NVQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679440199 (53h26m25.11052033s) [noise=true , address=104.36.82.36:4638 , piece_id=XUABMR23MOZG5QXDR2P3R5BSO5HHRZOP5QVSLKIZADSOZGZZIZNQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679713732 (53h26m25.110793863s) [noise=true , address=104.36.82.37:4663 , piece_id=3IGY7SFDVZYNVVGICK3C2XRBA5LRW5OQVZDFKODY5LKW57HWUR7A, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679926259 (53h26m25.11100639s) [noise=true , address=104.36.82.60:4348 , piece_id=Z4WXOGSIHUUIZPFDCJYA7ACUQXPXZWCT5QH7QLXR65T6AP2HIBIQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680126402 (53h26m25.111206533s) [noise=true , address=104.36.82.56:4238 , piece_id=7YLBQEL63SEYD2OILQ2PRKEP7U46PT4PSPAASSV5DJPAYTESOJ5A, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680326895 (53h26m25.111407026s) [noise=true , address=104.36.82.8:4995 , piece_id=G3K3LCGGAPEURR4G7HCBRM36ZBI4Q7MY2X5YT4CJLS2LRDRKXE7Q, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680530465 (53h26m25.111610596s) [noise=true , address=104.36.82.9:4030 , piece_id=7NIT67HJX2R7FUFLSSW36R54PJFKCGAKM5YJL6QWTO46UAY7GI5A, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680733313 (53h26m25.111813444s) [noise=true , address=104.36.82.16:4310 , piece_id=OXZLVTO5EASBCCHBSGFSHQSF2AI2U2XI6ENEIZPWN6T5YM7I2SLQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.681408626 (53h26m25.112488777s) [noise=true , address=104.36.82.44:4888 , piece_id=DPVGIMFES6RWUHO5OKRYJVX2HEDBBQJZHYYELVLMCBOWUKP6IKJA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.681726623 (53h26m25.112806755s) [noise=true , address=104.36.82.39:4746 , piece_id=UY4KKODNGNKAFOULMRSDCYJDT5HW7C6KYUD4EKZY3SBLXZRALOTQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:31.876216927 (53h26m26.307297058s) [upload=55724]: Next returned nil segment Aug 10 22:52:31.876232165 (53h26m26.307312296s) [upload=55724]: Waiting for error group managing segments... Aug 10 22:52:32.182969622 (53h26m26.614049743s) [noise=true , address=64.49.13.169:4713 , piece_id=7NZUZRKMLSKQ2YCASM264IOLWXRM5ZLQYAP53HL5BD52BLBCAEPA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.631903297 (53h26m27.062983428s) [noise=true , address=104.36.82.46:4507 , piece_id=B4NHEAVESRXDYUIOJ7A4WKZJOK4NPFT2JADD3JZXBPDVULAUEPIA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.652429340 (53h26m27.083509471s) [noise=true , address=104.36.82.34:4564 , piece_id=AVAXGPPYXYSSZIIG47NV4RISUD4RPP7S6KTYF4QYSAPDM3NYOBDA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.695199823 (53h26m27.126279954s) [noise=true , address=104.36.82.16:4310 , piece_id=OXZLVTO5EASBCCHBSGFSHQSF2AI2U2XI6ENEIZPWN6T5YM7I2SLQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.740711713 (53h26m27.171791844s) [noise=true , address=104.36.82.36:4638 , piece_id=XUABMR23MOZG5QXDR2P3R5BSO5HHRZOP5QVSLKIZADSOZGZZIZNQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.776799260 (53h26m27.207879391s) [noise=true , address=45.91.116.14:4221 , piece_id=GAKJS3G242VOGC66CVIKCPTX7ITXYM2SQFVJPUAVJHJI36DIZ4BQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.823959072 (53h26m27.255039203s) [noise=true , address=104.36.82.44:4888 , piece_id=DPVGIMFES6RWUHO5OKRYJVX2HEDBBQJZHYYELVLMCBOWUKP6IKJA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.873921155 (53h26m27.305001286s) [noise=true , address=104.36.82.35:4583 , piece_id=IXCJX5XZWILRXQMF24MUABV3TV2TW23R3T73XO563FKSTCDCYIYQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.886211631 (53h26m27.317291762s) [noise=true , address=104.36.82.45:4913 , piece_id=QA35NTBOIGIZWNNTBFZXECHK4SQUHBVAHWPCXNAH2YAAERZEJHMQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.886240236 (53h26m27.317320367s) [noise=true , address=104.36.82.47:4549 , piece_id=IOXOLG2NU2H5CZIEXKPLH4EIJVKK4GX66SZQPDSCNPPKSOGFTUYA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:33.361231516 (53h26m27.792311647s) [noise=true , address=104.36.82.9:4030 , piece_id=7NIT67HJX2R7FUFLSSW36R54PJFKCGAKM5YJL6QWTO46UAY7GI5A, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.095423782 (53h26m28.526503913s) [noise=true , address=104.36.82.20:4453 , piece_id=LHTZE3WPC3N5RCNUZKGMPKW3EYUR2FPVB2G6DFL6PL4ZECRYBAIA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.191766598 (53h26m28.622846729s) [noise=true , address=104.36.82.61:4378 , piece_id=KHNTUED6WVXAB6A7NZOE3JIUMY6FPQOJECGFWQG6ATWVCKMW2NVQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.957234535 (53h26m29.388314666s) [noise=true , address=104.36.82.39:4746 , piece_id=UY4KKODNGNKAFOULMRSDCYJDT5HW7C6KYUD4EKZY3SBLXZRALOTQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.957270263 (53h26m29.388350384s) [seg_pos={0 0}, upload=55724]: Segment reached optimal threshold of 14 pieces. Aug 10 22:52:34.957519320 (53h26m29.388599441s) [noise=true , address=104.36.82.60:4348 , piece_id=Z4WXOGSIHUUIZPFDCJYA7ACUQXPXZWCT5QH7QLXR65T6AP2HIBIQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: context canceled; context canceled Aug 10 22:52:34.957638617 (53h26m29.388718748s) [noise=true , address=104.36.82.37:4663 , piece_id=3IGY7SFDVZYNVVGICK3C2XRBA5LRW5OQVZDFKODY5LKW57HWUR7A, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: read tcp 188.166.225.170:38236->104.36.82.37:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:52:34.957696528 (53h26m29.388776659s) [noise=true , address=104.36.82.56:4238 , piece_id=7YLBQEL63SEYD2OILQ2PRKEP7U46PT4PSPAASSV5DJPAYTESOJ5A, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5): protocol: write tcp 188.166.225.170:55512->104.36.82.56:4238: use of closed network connection; write tcp 188.166.225.170:55512->104.36.82.56:4238: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55512->104.36.82.56:4238: use of closed network connection Aug 10 22:52:34.957729272 (53h26m29.388809393s) [noise=true , address=104.36.82.8:4995 , piece_id=G3K3LCGGAPEURR4G7HCBRM36ZBI4Q7MY2X5YT4CJLS2LRDRKXE7Q, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: rpc: dial tcp 104.36.82.8:4995: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:52:34.957748227 (53h26m29.388828348s) [seg_pos={0 0}, upload=55724]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:52:34.957846646 (53h26m29.388926767s) [upload=55724]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:52:34.957851144 (53h26m29.388931275s) [upload=55724]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:52:35.306840333 (53h26m29.737920484s) [upload=55724]: Done uploading segments... Aug 10 22:52:35.306857145 (53h26m29.737937276s) [upload=55724]: Upload finished. err: <nil> Duration: 7.680872406s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 7.320901549s FAIL (2025-08-10 22:53:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 7.320925424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.858571514s FAIL (2025-08-10 22:53:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.858619986s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.544730774s FAIL (2025-08-10 22:53:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 6.544753436s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 7.685397955s FAIL (2025-08-10 22:53:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 7.685427592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.178417089s FAIL (2025-08-10 22:53:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 6.178446165s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.098527099s FAIL (2025-08-10 22:53:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.09854844s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 6.798814621s FAIL (2025-08-10 22:53:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.798837175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [........X..X...............X.......X.X.....X.X...........XX.] duration: 8.836877624s FAIL (2025-08-10 22:53:00 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 10 22:52:30.226093181 (53h26m24.657173312s) [upload=55724]: Starting upload Aug 10 22:52:30.226146663 (53h26m24.657226794s) [upload=55724]: Done starting upload Aug 10 22:52:30.226228670 (53h26m24.657308801s) [upload=55724]: Uploading segments... Aug 10 22:52:30.226383958 (53h26m24.657464089s) [upload=55724]: Got next segment. Inline: false Aug 10 22:52:30.226415327 (53h26m24.657495458s) [upload=55724]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:52:30.226421208 (53h26m24.65750134s) [upload=55724]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:52:30.677132253 (53h26m25.108212384s) [seg_pos={0 0}, upload=55724]: Begin upload segment... Aug 10 22:52:30.677190715 (53h26m25.108270846s) [seg_pos={0 0}, upload=55724]: Done begin upload segment. Aug 10 22:52:30.677225091 (53h26m25.108305212s) [noise=true , address=104.36.82.46:4507 , piece_id=B4NHEAVESRXDYUIOJ7A4WKZJOK4NPFT2JADD3JZXBPDVULAUEPIA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.677252474 (53h26m25.108332605s) [noise=true , address=45.91.116.14:4221 , piece_id=GAKJS3G242VOGC66CVIKCPTX7ITXYM2SQFVJPUAVJHJI36DIZ4BQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.677613114 (53h26m25.108693245s) [noise=true , address=104.36.82.45:4913 , piece_id=QA35NTBOIGIZWNNTBFZXECHK4SQUHBVAHWPCXNAH2YAAERZEJHMQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678313253 (53h26m25.109393384s) [noise=true , address=64.49.13.169:4713 , piece_id=7NZUZRKMLSKQ2YCASM264IOLWXRM5ZLQYAP53HL5BD52BLBCAEPA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678331708 (53h26m25.109411839s) [noise=true , address=104.36.82.35:4583 , piece_id=IXCJX5XZWILRXQMF24MUABV3TV2TW23R3T73XO563FKSTCDCYIYQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678613808 (53h26m25.109693939s) [noise=true , address=104.36.82.34:4564 , piece_id=AVAXGPPYXYSSZIIG47NV4RISUD4RPP7S6KTYF4QYSAPDM3NYOBDA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678652442 (53h26m25.109732573s) [noise=true , address=104.36.82.47:4549 , piece_id=IOXOLG2NU2H5CZIEXKPLH4EIJVKK4GX66SZQPDSCNPPKSOGFTUYA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678866331 (53h26m25.109946462s) [noise=true , address=104.36.82.20:4453 , piece_id=LHTZE3WPC3N5RCNUZKGMPKW3EYUR2FPVB2G6DFL6PL4ZECRYBAIA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679169000 (53h26m25.110249131s) [noise=true , address=104.36.82.61:4378 , piece_id=KHNTUED6WVXAB6A7NZOE3JIUMY6FPQOJECGFWQG6ATWVCKMW2NVQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679440199 (53h26m25.11052033s) [noise=true , address=104.36.82.36:4638 , piece_id=XUABMR23MOZG5QXDR2P3R5BSO5HHRZOP5QVSLKIZADSOZGZZIZNQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679713732 (53h26m25.110793863s) [noise=true , address=104.36.82.37:4663 , piece_id=3IGY7SFDVZYNVVGICK3C2XRBA5LRW5OQVZDFKODY5LKW57HWUR7A, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679926259 (53h26m25.11100639s) [noise=true , address=104.36.82.60:4348 , piece_id=Z4WXOGSIHUUIZPFDCJYA7ACUQXPXZWCT5QH7QLXR65T6AP2HIBIQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680126402 (53h26m25.111206533s) [noise=true , address=104.36.82.56:4238 , piece_id=7YLBQEL63SEYD2OILQ2PRKEP7U46PT4PSPAASSV5DJPAYTESOJ5A, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680326895 (53h26m25.111407026s) [noise=true , address=104.36.82.8:4995 , piece_id=G3K3LCGGAPEURR4G7HCBRM36ZBI4Q7MY2X5YT4CJLS2LRDRKXE7Q, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680530465 (53h26m25.111610596s) [noise=true , address=104.36.82.9:4030 , piece_id=7NIT67HJX2R7FUFLSSW36R54PJFKCGAKM5YJL6QWTO46UAY7GI5A, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680733313 (53h26m25.111813444s) [noise=true , address=104.36.82.16:4310 , piece_id=OXZLVTO5EASBCCHBSGFSHQSF2AI2U2XI6ENEIZPWN6T5YM7I2SLQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.681408626 (53h26m25.112488777s) [noise=true , address=104.36.82.44:4888 , piece_id=DPVGIMFES6RWUHO5OKRYJVX2HEDBBQJZHYYELVLMCBOWUKP6IKJA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.681726623 (53h26m25.112806755s) [noise=true , address=104.36.82.39:4746 , piece_id=UY4KKODNGNKAFOULMRSDCYJDT5HW7C6KYUD4EKZY3SBLXZRALOTQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:31.876216927 (53h26m26.307297058s) [upload=55724]: Next returned nil segment Aug 10 22:52:31.876232165 (53h26m26.307312296s) [upload=55724]: Waiting for error group managing segments... Aug 10 22:52:32.182969622 (53h26m26.614049743s) [noise=true , address=64.49.13.169:4713 , piece_id=7NZUZRKMLSKQ2YCASM264IOLWXRM5ZLQYAP53HL5BD52BLBCAEPA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.631903297 (53h26m27.062983428s) [noise=true , address=104.36.82.46:4507 , piece_id=B4NHEAVESRXDYUIOJ7A4WKZJOK4NPFT2JADD3JZXBPDVULAUEPIA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.652429340 (53h26m27.083509471s) [noise=true , address=104.36.82.34:4564 , piece_id=AVAXGPPYXYSSZIIG47NV4RISUD4RPP7S6KTYF4QYSAPDM3NYOBDA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.695199823 (53h26m27.126279954s) [noise=true , address=104.36.82.16:4310 , piece_id=OXZLVTO5EASBCCHBSGFSHQSF2AI2U2XI6ENEIZPWN6T5YM7I2SLQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.740711713 (53h26m27.171791844s) [noise=true , address=104.36.82.36:4638 , piece_id=XUABMR23MOZG5QXDR2P3R5BSO5HHRZOP5QVSLKIZADSOZGZZIZNQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.776799260 (53h26m27.207879391s) [noise=true , address=45.91.116.14:4221 , piece_id=GAKJS3G242VOGC66CVIKCPTX7ITXYM2SQFVJPUAVJHJI36DIZ4BQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.823959072 (53h26m27.255039203s) [noise=true , address=104.36.82.44:4888 , piece_id=DPVGIMFES6RWUHO5OKRYJVX2HEDBBQJZHYYELVLMCBOWUKP6IKJA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.873921155 (53h26m27.305001286s) [noise=true , address=104.36.82.35:4583 , piece_id=IXCJX5XZWILRXQMF24MUABV3TV2TW23R3T73XO563FKSTCDCYIYQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.886211631 (53h26m27.317291762s) [noise=true , address=104.36.82.45:4913 , piece_id=QA35NTBOIGIZWNNTBFZXECHK4SQUHBVAHWPCXNAH2YAAERZEJHMQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.886240236 (53h26m27.317320367s) [noise=true , address=104.36.82.47:4549 , piece_id=IOXOLG2NU2H5CZIEXKPLH4EIJVKK4GX66SZQPDSCNPPKSOGFTUYA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:33.361231516 (53h26m27.792311647s) [noise=true , address=104.36.82.9:4030 , piece_id=7NIT67HJX2R7FUFLSSW36R54PJFKCGAKM5YJL6QWTO46UAY7GI5A, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.095423782 (53h26m28.526503913s) [noise=true , address=104.36.82.20:4453 , piece_id=LHTZE3WPC3N5RCNUZKGMPKW3EYUR2FPVB2G6DFL6PL4ZECRYBAIA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.191766598 (53h26m28.622846729s) [noise=true , address=104.36.82.61:4378 , piece_id=KHNTUED6WVXAB6A7NZOE3JIUMY6FPQOJECGFWQG6ATWVCKMW2NVQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.957234535 (53h26m29.388314666s) [noise=true , address=104.36.82.39:4746 , piece_id=UY4KKODNGNKAFOULMRSDCYJDT5HW7C6KYUD4EKZY3SBLXZRALOTQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.957270263 (53h26m29.388350384s) [seg_pos={0 0}, upload=55724]: Segment reached optimal threshold of 14 pieces. Aug 10 22:52:34.957519320 (53h26m29.388599441s) [noise=true , address=104.36.82.60:4348 , piece_id=Z4WXOGSIHUUIZPFDCJYA7ACUQXPXZWCT5QH7QLXR65T6AP2HIBIQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: context canceled; context canceled Aug 10 22:52:34.957638617 (53h26m29.388718748s) [noise=true , address=104.36.82.37:4663 , piece_id=3IGY7SFDVZYNVVGICK3C2XRBA5LRW5OQVZDFKODY5LKW57HWUR7A, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: read tcp 188.166.225.170:38236->104.36.82.37:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:52:34.957696528 (53h26m29.388776659s) [noise=true , address=104.36.82.56:4238 , piece_id=7YLBQEL63SEYD2OILQ2PRKEP7U46PT4PSPAASSV5DJPAYTESOJ5A, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5): protocol: write tcp 188.166.225.170:55512->104.36.82.56:4238: use of closed network connection; write tcp 188.166.225.170:55512->104.36.82.56:4238: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55512->104.36.82.56:4238: use of closed network connection Aug 10 22:52:34.957729272 (53h26m29.388809393s) [noise=true , address=104.36.82.8:4995 , piece_id=G3K3LCGGAPEURR4G7HCBRM36ZBI4Q7MY2X5YT4CJLS2LRDRKXE7Q, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: rpc: dial tcp 104.36.82.8:4995: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:52:34.957748227 (53h26m29.388828348s) [seg_pos={0 0}, upload=55724]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:52:34.957846646 (53h26m29.388926767s) [upload=55724]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:52:34.957851144 (53h26m29.388931275s) [upload=55724]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:52:35.306840333 (53h26m29.737920484s) [upload=55724]: Done uploading segments... Aug 10 22:52:35.306857145 (53h26m29.737937276s) [upload=55724]: Upload finished. err: <nil> Duration: 8.836899375s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 7.861217223s FAIL (2025-08-10 22:52:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 7.86124615s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 7.13306275s FAIL (2025-08-10 22:52:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 7.133087637s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........X..X...............X.......X.X.....X.X...........XX] duration: 6.398923537s FAIL (2025-08-10 22:52:35 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 10 22:52:30.226093181 (53h26m24.657173312s) [upload=55724]: Starting upload Aug 10 22:52:30.226146663 (53h26m24.657226794s) [upload=55724]: Done starting upload Aug 10 22:52:30.226228670 (53h26m24.657308801s) [upload=55724]: Uploading segments... Aug 10 22:52:30.226383958 (53h26m24.657464089s) [upload=55724]: Got next segment. Inline: false Aug 10 22:52:30.226415327 (53h26m24.657495458s) [upload=55724]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:52:30.226421208 (53h26m24.65750134s) [upload=55724]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:52:30.677132253 (53h26m25.108212384s) [seg_pos={0 0}, upload=55724]: Begin upload segment... Aug 10 22:52:30.677190715 (53h26m25.108270846s) [seg_pos={0 0}, upload=55724]: Done begin upload segment. Aug 10 22:52:30.677225091 (53h26m25.108305212s) [noise=true , address=104.36.82.46:4507 , piece_id=B4NHEAVESRXDYUIOJ7A4WKZJOK4NPFT2JADD3JZXBPDVULAUEPIA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.677252474 (53h26m25.108332605s) [noise=true , address=45.91.116.14:4221 , piece_id=GAKJS3G242VOGC66CVIKCPTX7ITXYM2SQFVJPUAVJHJI36DIZ4BQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.677613114 (53h26m25.108693245s) [noise=true , address=104.36.82.45:4913 , piece_id=QA35NTBOIGIZWNNTBFZXECHK4SQUHBVAHWPCXNAH2YAAERZEJHMQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678313253 (53h26m25.109393384s) [noise=true , address=64.49.13.169:4713 , piece_id=7NZUZRKMLSKQ2YCASM264IOLWXRM5ZLQYAP53HL5BD52BLBCAEPA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678331708 (53h26m25.109411839s) [noise=true , address=104.36.82.35:4583 , piece_id=IXCJX5XZWILRXQMF24MUABV3TV2TW23R3T73XO563FKSTCDCYIYQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678613808 (53h26m25.109693939s) [noise=true , address=104.36.82.34:4564 , piece_id=AVAXGPPYXYSSZIIG47NV4RISUD4RPP7S6KTYF4QYSAPDM3NYOBDA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678652442 (53h26m25.109732573s) [noise=true , address=104.36.82.47:4549 , piece_id=IOXOLG2NU2H5CZIEXKPLH4EIJVKK4GX66SZQPDSCNPPKSOGFTUYA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.678866331 (53h26m25.109946462s) [noise=true , address=104.36.82.20:4453 , piece_id=LHTZE3WPC3N5RCNUZKGMPKW3EYUR2FPVB2G6DFL6PL4ZECRYBAIA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679169000 (53h26m25.110249131s) [noise=true , address=104.36.82.61:4378 , piece_id=KHNTUED6WVXAB6A7NZOE3JIUMY6FPQOJECGFWQG6ATWVCKMW2NVQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679440199 (53h26m25.11052033s) [noise=true , address=104.36.82.36:4638 , piece_id=XUABMR23MOZG5QXDR2P3R5BSO5HHRZOP5QVSLKIZADSOZGZZIZNQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679713732 (53h26m25.110793863s) [noise=true , address=104.36.82.37:4663 , piece_id=3IGY7SFDVZYNVVGICK3C2XRBA5LRW5OQVZDFKODY5LKW57HWUR7A, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.679926259 (53h26m25.11100639s) [noise=true , address=104.36.82.60:4348 , piece_id=Z4WXOGSIHUUIZPFDCJYA7ACUQXPXZWCT5QH7QLXR65T6AP2HIBIQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680126402 (53h26m25.111206533s) [noise=true , address=104.36.82.56:4238 , piece_id=7YLBQEL63SEYD2OILQ2PRKEP7U46PT4PSPAASSV5DJPAYTESOJ5A, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680326895 (53h26m25.111407026s) [noise=true , address=104.36.82.8:4995 , piece_id=G3K3LCGGAPEURR4G7HCBRM36ZBI4Q7MY2X5YT4CJLS2LRDRKXE7Q, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680530465 (53h26m25.111610596s) [noise=true , address=104.36.82.9:4030 , piece_id=7NIT67HJX2R7FUFLSSW36R54PJFKCGAKM5YJL6QWTO46UAY7GI5A, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.680733313 (53h26m25.111813444s) [noise=true , address=104.36.82.16:4310 , piece_id=OXZLVTO5EASBCCHBSGFSHQSF2AI2U2XI6ENEIZPWN6T5YM7I2SLQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.681408626 (53h26m25.112488777s) [noise=true , address=104.36.82.44:4888 , piece_id=DPVGIMFES6RWUHO5OKRYJVX2HEDBBQJZHYYELVLMCBOWUKP6IKJA, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:30.681726623 (53h26m25.112806755s) [noise=true , address=104.36.82.39:4746 , piece_id=UY4KKODNGNKAFOULMRSDCYJDT5HW7C6KYUD4EKZY3SBLXZRALOTQ, seg_pos={0 0}, upload=55724]: Uploading piece... Aug 10 22:52:31.876216927 (53h26m26.307297058s) [upload=55724]: Next returned nil segment Aug 10 22:52:31.876232165 (53h26m26.307312296s) [upload=55724]: Waiting for error group managing segments... Aug 10 22:52:32.182969622 (53h26m26.614049743s) [noise=true , address=64.49.13.169:4713 , piece_id=7NZUZRKMLSKQ2YCASM264IOLWXRM5ZLQYAP53HL5BD52BLBCAEPA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.631903297 (53h26m27.062983428s) [noise=true , address=104.36.82.46:4507 , piece_id=B4NHEAVESRXDYUIOJ7A4WKZJOK4NPFT2JADD3JZXBPDVULAUEPIA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.652429340 (53h26m27.083509471s) [noise=true , address=104.36.82.34:4564 , piece_id=AVAXGPPYXYSSZIIG47NV4RISUD4RPP7S6KTYF4QYSAPDM3NYOBDA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.695199823 (53h26m27.126279954s) [noise=true , address=104.36.82.16:4310 , piece_id=OXZLVTO5EASBCCHBSGFSHQSF2AI2U2XI6ENEIZPWN6T5YM7I2SLQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.740711713 (53h26m27.171791844s) [noise=true , address=104.36.82.36:4638 , piece_id=XUABMR23MOZG5QXDR2P3R5BSO5HHRZOP5QVSLKIZADSOZGZZIZNQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.776799260 (53h26m27.207879391s) [noise=true , address=45.91.116.14:4221 , piece_id=GAKJS3G242VOGC66CVIKCPTX7ITXYM2SQFVJPUAVJHJI36DIZ4BQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.823959072 (53h26m27.255039203s) [noise=true , address=104.36.82.44:4888 , piece_id=DPVGIMFES6RWUHO5OKRYJVX2HEDBBQJZHYYELVLMCBOWUKP6IKJA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.873921155 (53h26m27.305001286s) [noise=true , address=104.36.82.35:4583 , piece_id=IXCJX5XZWILRXQMF24MUABV3TV2TW23R3T73XO563FKSTCDCYIYQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.886211631 (53h26m27.317291762s) [noise=true , address=104.36.82.45:4913 , piece_id=QA35NTBOIGIZWNNTBFZXECHK4SQUHBVAHWPCXNAH2YAAERZEJHMQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:32.886240236 (53h26m27.317320367s) [noise=true , address=104.36.82.47:4549 , piece_id=IOXOLG2NU2H5CZIEXKPLH4EIJVKK4GX66SZQPDSCNPPKSOGFTUYA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:33.361231516 (53h26m27.792311647s) [noise=true , address=104.36.82.9:4030 , piece_id=7NIT67HJX2R7FUFLSSW36R54PJFKCGAKM5YJL6QWTO46UAY7GI5A, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.095423782 (53h26m28.526503913s) [noise=true , address=104.36.82.20:4453 , piece_id=LHTZE3WPC3N5RCNUZKGMPKW3EYUR2FPVB2G6DFL6PL4ZECRYBAIA, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.191766598 (53h26m28.622846729s) [noise=true , address=104.36.82.61:4378 , piece_id=KHNTUED6WVXAB6A7NZOE3JIUMY6FPQOJECGFWQG6ATWVCKMW2NVQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.957234535 (53h26m29.388314666s) [noise=true , address=104.36.82.39:4746 , piece_id=UY4KKODNGNKAFOULMRSDCYJDT5HW7C6KYUD4EKZY3SBLXZRALOTQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: <nil> Aug 10 22:52:34.957270263 (53h26m29.388350384s) [seg_pos={0 0}, upload=55724]: Segment reached optimal threshold of 14 pieces. Aug 10 22:52:34.957519320 (53h26m29.388599441s) [noise=true , address=104.36.82.60:4348 , piece_id=Z4WXOGSIHUUIZPFDCJYA7ACUQXPXZWCT5QH7QLXR65T6AP2HIBIQ, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: context canceled; context canceled Aug 10 22:52:34.957638617 (53h26m29.388718748s) [noise=true , address=104.36.82.37:4663 , piece_id=3IGY7SFDVZYNVVGICK3C2XRBA5LRW5OQVZDFKODY5LKW57HWUR7A, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: read tcp 188.166.225.170:38236->104.36.82.37:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:52:34.957696528 (53h26m29.388776659s) [noise=true , address=104.36.82.56:4238 , piece_id=7YLBQEL63SEYD2OILQ2PRKEP7U46PT4PSPAASSV5DJPAYTESOJ5A, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5): protocol: write tcp 188.166.225.170:55512->104.36.82.56:4238: use of closed network connection; write tcp 188.166.225.170:55512->104.36.82.56:4238: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55512->104.36.82.56:4238: use of closed network connection Aug 10 22:52:34.957729272 (53h26m29.388809393s) [noise=true , address=104.36.82.8:4995 , piece_id=G3K3LCGGAPEURR4G7HCBRM36ZBI4Q7MY2X5YT4CJLS2LRDRKXE7Q, seg_pos={0 0}, upload=55724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: rpc: dial tcp 104.36.82.8:4995: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:52:34.957748227 (53h26m29.388828348s) [seg_pos={0 0}, upload=55724]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:52:34.957846646 (53h26m29.388926767s) [upload=55724]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:52:34.957851144 (53h26m29.388931275s) [upload=55724]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:52:35.306840333 (53h26m29.737920484s) [upload=55724]: Done uploading segments... Aug 10 22:52:35.306857145 (53h26m29.737937276s) [upload=55724]: Upload finished. err: <nil> Duration: 6.399013499s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 8.924829534s FAIL (2025-08-10 22:52:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 8.924851106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X...............................X...........................] duration: 2.03556475s FAIL (2025-08-10 22:52:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.035589327s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 7.076592446s FAIL (2025-08-10 22:51:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 7.076621102s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X...............................X..........................] duration: 2.07196646s FAIL (2025-08-10 22:51:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.071995445s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 6.788713901s FAIL (2025-08-10 22:51:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.788741464s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.547837167s FAIL (2025-08-10 22:51:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 6.547861874s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........X..X...............X.......X.X.....X.X...........X] duration: 4.949190292s FAIL (2025-08-10 22:51:34 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 10 22:51:30.238852406 (53h25m24.669932537s) [upload=55705]: Starting upload Aug 10 22:51:30.238896109 (53h25m24.66997624s) [upload=55705]: Done starting upload Aug 10 22:51:30.238981943 (53h25m24.670062104s) [upload=55705]: Uploading segments... Aug 10 22:51:30.239088497 (53h25m24.670168648s) [upload=55705]: Got next segment. Inline: false Aug 10 22:51:30.239128765 (53h25m24.670208896s) [upload=55705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:51:30.239134997 (53h25m24.670215138s) [upload=55705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:51:30.715843894 (53h25m25.146924026s) [seg_pos={0 0}, upload=55705]: Begin upload segment... Aug 10 22:51:30.715889001 (53h25m25.146969132s) [seg_pos={0 0}, upload=55705]: Done begin upload segment. Aug 10 22:51:30.715902297 (53h25m25.146982428s) [noise=true , address=104.36.82.44:4883 , piece_id=5TJUPJYQR4GLT46H3MUGNLCM5SN4I2N722X7BRCM2IVXEGIZLQ4Q, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.715910913 (53h25m25.146991044s) [noise=true , address=104.36.82.5:4869 , piece_id=TQWUT26ZW6BEUWKC2DWRTQ7OVCFDH52HKIR3VDGIE4ZS6OPK6GQA, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.716250012 (53h25m25.147330143s) [noise=true , address=104.36.82.47:4521 , piece_id=2VEB6NO7GKXLUQGA3T77RPLBNUHYMTSPKGL3DLCUCORJL7G5ZRYQ, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.716263518 (53h25m25.147343649s) [noise=true , address=45.91.116.6:4998 , piece_id=3MQMHMGHZ44IKVSCZRKANHVEZX7XOY4AC25HTDUL6ATFAT7BGCTQ, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.716631512 (53h25m25.147711633s) [noise=true , address=64.49.13.170:4728 , piece_id=O6ZD6RRRL7XTK4NAN5PKBCDRDUKRQ2NKZK25YDPJOUKVZW4ZFFKA, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.716907820 (53h25m25.147987951s) [noise=true , address=104.36.82.6:4902 , piece_id=ST42IH7SYD42RH6V32SZDOQ2MIKVUYU4N6ROF5VJ3JDZ5L7WBWNQ, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.717231059 (53h25m25.14831118s) [noise=true , address=193.149.164.143:4697 , piece_id=O5A7UR4ZOBZCSRCVE337UGGJJWW64ERX73YFZ2KRC7BYZAFR3LVQ, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.717466198 (53h25m25.148546329s) [noise=true , address=104.36.82.9:4014 , piece_id=UZK3DEWAXIJ6Z7BT7KZUFLYSLN3VQED5W44EHLYL3UI4VIYEGFJQ, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.717690548 (53h25m25.14877068s) [noise=true , address=79.127.201.215:4346 , piece_id=OMMPSCT7SY42MMAK7UVR3GRC6MTCRTGES2KZCJ7ESDGYACZWYTTQ, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.717941519 (53h25m25.14902164s) [noise=true , address=104.36.82.37:4677 , piece_id=2DSKVX2ER63QHPWNBQL2THAEMPEFPE6VR6YG45N5VHXGYUANMCPA, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.718158323 (53h25m25.149238455s) [noise=true , address=79.127.219.48:4088 , piece_id=2W7J24QQMZAJAIY5H5XMU5BSHDCFFHAZNSF6GPYYWQ2L3HNYFGIQ, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.718373134 (53h25m25.149453265s) [noise=true , address=104.36.82.62:4409 , piece_id=XN53DXVXNJMLVWIII3KGXYII45DFSNI4ZTYTHROTBKGGUXY54ROQ, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.718561254 (53h25m25.149641375s) [noise=true , address=104.36.82.35:4612 , piece_id=HQX65DZ7MAD6ZFZ442Q7ZAMT4N2JBTZXFXJKOITXELT4KQZTBEDA, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.718757820 (53h25m25.149837951s) [noise=true , address=104.36.82.59:4331 , piece_id=EUDTUJQYGUBITOGPSBRNTH2QTCVBKKOCWC7JCOME4WCZEMDJXMJQ, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.718949697 (53h25m25.150029828s) [noise=true , address=156.146.43.225:4440 , piece_id=VSFSSK4VRUIUTWAZYE2JPOSZT2SB5ISVSDM76MI35ZEZVPT4EDFQ, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.719152596 (53h25m25.150232717s) [noise=true , address=104.36.82.60:4346 , piece_id=TCB7FKDU7X3RGPPLQ5IYX6P75FYHE3GPMFN277E3HSJWQB5BWTCQ, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.719438362 (53h25m25.150518483s) [noise=true , address=104.36.82.58:4292 , piece_id=ALEP2RJ5MAF6YKUQMG4VASMCCVUZSW5IRQK2M7VNOTSF6MY556BA, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:30.719641731 (53h25m25.150721852s) [noise=true , address=104.36.82.18:4386 , piece_id=V3LIG7B4FLZSB5DRJK24PZSGFQQ745VJ5M27FPTQ6FDFGM7P7FTA, seg_pos={0 0}, upload=55705]: Uploading piece... Aug 10 22:51:31.887181139 (53h25m26.31826128s) [upload=55705]: Next returned nil segment Aug 10 22:51:31.887200756 (53h25m26.318280887s) [upload=55705]: Waiting for error group managing segments... Aug 10 22:51:32.186108243 (53h25m26.617188374s) [noise=true , address=64.49.13.170:4728 , piece_id=O6ZD6RRRL7XTK4NAN5PKBCDRDUKRQ2NKZK25YDPJOUKVZW4ZFFKA, seg_pos={0 0}, upload=55705]: Done uploading piece. err: <nil> Aug 10 22:51:32.213255324 (53h25m26.644335455s) [noise=true , address=193.149.164.143:4697 , piece_id=O5A7UR4ZOBZCSRCVE337UGGJJWW64ERX73YFZ2KRC7BYZAFR3LVQ, seg_pos={0 0}, upload=55705]: Done uploading piece. err: <nil> Aug 10 22:51:32.615246842 (53h25m27.046326973s) [noise=true , address=79.127.219.48:4088 , piece_id=2W7J24QQMZAJAIY5H5XMU5BSHDCFFHAZNSF6GPYYWQ2L3HNYFGIQ, seg_pos={0 0}, upload=55705]: Done uploading piece. err: <nil> Aug 10 22:51:32.704108333 (53h25m27.135188474s) [noise=true , address=104.36.82.60:4346 , piece_id=TCB7FKDU7X3RGPPLQ5IYX6P75FYHE3GPMFN277E3HSJWQB5BWTCQ, seg_pos={0 0}, upload=55705]: Done uploading piece. err: <nil> Aug 10 22:51:32.707742868 (53h25m27.138822999s) [noise=true , address=79.127.201.215:4346 , piece_id=OMMPSCT7SY42MMAK7UVR3GRC6MTCRTGES2KZCJ7ESDGYACZWYTTQ, seg_pos={0 0}, upload=55705]: Done uploading piece. err: <nil> Aug 10 22:51:32.745051089 (53h25m27.17613123s) [noise=true , address=156.146.43.225:4440 , piece_id=VSFSSK4VRUIUTWAZYE2JPOSZT2SB5ISVSDM76MI35ZEZVPT4EDFQ, seg_pos={0 0}, upload=55705]: Done uploading piece. err: <nil> Aug 10 22:51:32.800123912 (53h25m27.231204043s) [noise=true , address=104.36.82.59:4331 , piece_id=EUDTUJQYGUBITOGPSBRNTH2QTCVBKKOCWC7JCOME4WCZEMDJXMJQ, seg_pos={0 0}, upload=55705]: Done uploading piece. err: <nil> Aug 10 22:51:32.828713993 (53h25m27.259794124s) [noise=true , address=104.36.82.18:4386 , piece_id=V3LIG7B4FLZSB5DRJK24PZSGFQQ745VJ5M27FPTQ6FDFGM7P7FTA, seg_pos={0 0}, upload=55705]: Done uploading piece. err: <nil> Aug 10 22:51:32.890581826 (53h25m27.321661957s) [noise=true , address=104.36.82.9:4014 , piece_id=UZK3DEWAXIJ6Z7BT7KZUFLYSLN3VQED5W44EHLYL3UI4VIYEGFJQ, seg_pos={0 0}, upload=55705]: Done uploading piece. err: <nil> Aug 10 22:51:32.914559854 (53h25m27.345639975s) [noise=true , address=104.36.82.5:4869 , piece_id=TQWUT26ZW6BEUWKC2DWRTQ7OVCFDH52HKIR3VDGIE4ZS6OPK6GQA, seg_pos={0 0}, upload=55705]: Done uploading piece. err: <nil> Aug 10 22:51:32.963640029 (53h25m27.39472015s) [noise=true , address=104.36.82.6:4902 , piece_id=ST42IH7SYD42RH6V32SZDOQ2MIKVUYU4N6ROF5VJ3JDZ5L7WBWNQ, seg_pos={0 0}, upload=55705]: Done uploading piece. err: <nil> Aug 10 22:51:33.083598509 (53h25m27.51467864s) [noise=true , address=45.91.116.6:4998 , piece_id=3MQMHMGHZ44IKVSCZRKANHVEZX7XOY4AC25HTDUL6ATFAT7BGCTQ, seg_pos={0 0}, upload=55705]: Done uploading piece. err: <nil> Aug 10 22:51:33.279678043 (53h25m27.710758174s) [noise=true , address=104.36.82.47:4521 , piece_id=2VEB6NO7GKXLUQGA3T77RPLBNUHYMTSPKGL3DLCUCORJL7G5ZRYQ, seg_pos={0 0}, upload=55705]: Done uploading piece. err: <nil> Aug 10 22:51:33.718341928 (53h25m28.149422069s) [noise=true , address=104.36.82.35:4612 , piece_id=HQX65DZ7MAD6ZFZ442Q7ZAMT4N2JBTZXFXJKOITXELT4KQZTBEDA, seg_pos={0 0}, upload=55705]: Done uploading piece. err: <nil> Aug 10 22:51:33.718373488 (53h25m28.149453619s) [seg_pos={0 0}, upload=55705]: Segment reached optimal threshold of 14 pieces. Aug 10 22:51:33.718633346 (53h25m28.149713477s) [noise=true , address=104.36.82.37:4677 , piece_id=2DSKVX2ER63QHPWNBQL2THAEMPEFPE6VR6YG45N5VHXGYUANMCPA, seg_pos={0 0}, upload=55705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QgDye4pcKhsndRP7tCxM44KZkSyRnhp219u8JnRHXX7jRutPv): protocol: expected piece hash; context canceled; EOF Aug 10 22:51:33.718738798 (53h25m28.149818929s) [noise=true , address=104.36.82.44:4883 , piece_id=5TJUPJYQR4GLT46H3MUGNLCM5SN4I2N722X7BRCM2IVXEGIZLQ4Q, seg_pos={0 0}, upload=55705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cx1avL4umDm2pzJS1rNZqeDZjZMWnMtNsyNTdMdkxhpa1ByUN): protocol: context canceled; context canceled Aug 10 22:51:33.718878315 (53h25m28.149958446s) [noise=true , address=104.36.82.62:4409 , piece_id=XN53DXVXNJMLVWIII3KGXYII45DFSNI4ZTYTHROTBKGGUXY54ROQ, seg_pos={0 0}, upload=55705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:49328->104.36.82.62:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:51:33.718907050 (53h25m28.149987201s) [noise=true , address=104.36.82.58:4292 , piece_id=ALEP2RJ5MAF6YKUQMG4VASMCCVUZSW5IRQK2M7VNOTSF6MY556BA, seg_pos={0 0}, upload=55705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JeDjqgszkkq2asUBCL5oTvH6X21TnWkkmqGEEzBbz1YctdDXX): protocol: read tcp 188.166.225.170:35138->104.36.82.58:4292: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:51:33.718922539 (53h25m28.15000267s) [seg_pos={0 0}, upload=55705]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:51:33.718998795 (53h25m28.150078946s) [upload=55705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:51:33.719004976 (53h25m28.150085138s) [upload=55705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:51:34.055009691 (53h25m28.486089852s) [upload=55705]: Done uploading segments... Aug 10 22:51:34.055033166 (53h25m28.486113327s) [upload=55705]: Upload finished. err: <nil> Duration: 4.949214327s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X..X...............X.......X.X.....X.X...........] duration: 7.769373931s FAIL (2025-08-10 22:51:19 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 10 22:42:04.151608101 (53h15m58.582688232s) [upload=55545]: Starting upload Aug 10 22:42:04.151655141 (53h15m58.582735272s) [upload=55545]: Done starting upload Aug 10 22:42:04.151712230 (53h15m58.582792351s) [upload=55545]: Uploading segments... Aug 10 22:42:04.151832401 (53h15m58.582912532s) [upload=55545]: Got next segment. Inline: false Aug 10 22:42:04.151867177 (53h15m58.582947298s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:42:04.151871315 (53h15m58.582951446s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:42:04.596187241 (53h15m59.027267432s) [seg_pos={0 0}, upload=55545]: Begin upload segment... Aug 10 22:42:04.596256584 (53h15m59.027336715s) [seg_pos={0 0}, upload=55545]: Done begin upload segment. Aug 10 22:42:04.596302321 (53h15m59.027382462s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.596321418 (53h15m59.027401559s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597244444 (53h15m59.028324605s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597424017 (53h15m59.028504189s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597875963 (53h15m59.028956104s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598094571 (53h15m59.029174712s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598428159 (53h15m59.02950832s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598838364 (53h15m59.029918495s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599186390 (53h15m59.030266521s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599809602 (53h15m59.030889733s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599928190 (53h15m59.031008321s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600314469 (53h15m59.0313946s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600730284 (53h15m59.031810415s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.601867690 (53h15m59.032947821s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.602627183 (53h15m59.033707314s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605005474 (53h15m59.036085615s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605487957 (53h15m59.036568098s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605924182 (53h15m59.037004323s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:05.750652684 (53h16m0.181732815s) [upload=55545]: Next returned nil segment Aug 10 22:42:05.750671300 (53h16m0.181751431s) [upload=55545]: Waiting for error group managing segments... Aug 10 22:42:06.068503390 (53h16m0.499583521s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.069092899 (53h16m0.50017303s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.107858230 (53h16m0.538938361s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.120952784 (53h16m0.552032915s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.344626386 (53h16m0.775706517s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.472396183 (53h16m0.903476314s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.599585118 (53h16m1.030665249s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.602989492 (53h16m1.034069613s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.641577704 (53h16m1.072657835s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.660099663 (53h16m1.091179784s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.806051468 (53h16m1.237131599s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.835071855 (53h16m1.266151986s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.841828643 (53h16m1.272908764s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853893837 (53h16m1.284973968s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853915288 (53h16m1.284995419s) [seg_pos={0 0}, upload=55545]: Segment reached optimal threshold of 14 pieces. Aug 10 22:42:06.854012876 (53h16m1.285092997s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qhGin74NgTpgUFEFrMdhXaQC3v17PefWnKgLSotok56yRoxZx): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854066027 (53h16m1.285146158s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854148315 (53h16m1.285228446s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854157402 (53h16m1.285237533s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854194522 (53h16m1.285274654s) [seg_pos={0 0}, upload=55545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:42:06.854287050 (53h16m1.285367181s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:42:06.854291298 (53h16m1.285371429s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:42:07.702881346 (53h16m2.133961477s) [upload=55545]: Done uploading segments... Aug 10 22:42:07.702904620 (53h16m2.133984791s) [upload=55545]: Upload finished. err: <nil> Duration: 7.769404037s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 6.084949212s FAIL (2025-08-10 22:50:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.084980543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X...............................X.........................] duration: 1.832491615s FAIL (2025-08-10 22:50:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.832522405s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 7.032521354s FAIL (2025-08-10 22:50:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 7.032546453s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.477974897s FAIL (2025-08-10 22:50:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 6.478005416s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 6.637090386s FAIL (2025-08-10 22:50:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.637130504s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X..X...............X.......X.X.....X.X..........] duration: 6.280917012s FAIL (2025-08-10 22:50:11 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 10 22:42:04.151608101 (53h15m58.582688232s) [upload=55545]: Starting upload Aug 10 22:42:04.151655141 (53h15m58.582735272s) [upload=55545]: Done starting upload Aug 10 22:42:04.151712230 (53h15m58.582792351s) [upload=55545]: Uploading segments... Aug 10 22:42:04.151832401 (53h15m58.582912532s) [upload=55545]: Got next segment. Inline: false Aug 10 22:42:04.151867177 (53h15m58.582947298s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:42:04.151871315 (53h15m58.582951446s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:42:04.596187241 (53h15m59.027267432s) [seg_pos={0 0}, upload=55545]: Begin upload segment... Aug 10 22:42:04.596256584 (53h15m59.027336715s) [seg_pos={0 0}, upload=55545]: Done begin upload segment. Aug 10 22:42:04.596302321 (53h15m59.027382462s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.596321418 (53h15m59.027401559s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597244444 (53h15m59.028324605s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597424017 (53h15m59.028504189s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597875963 (53h15m59.028956104s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598094571 (53h15m59.029174712s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598428159 (53h15m59.02950832s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598838364 (53h15m59.029918495s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599186390 (53h15m59.030266521s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599809602 (53h15m59.030889733s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599928190 (53h15m59.031008321s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600314469 (53h15m59.0313946s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600730284 (53h15m59.031810415s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.601867690 (53h15m59.032947821s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.602627183 (53h15m59.033707314s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605005474 (53h15m59.036085615s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605487957 (53h15m59.036568098s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605924182 (53h15m59.037004323s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:05.750652684 (53h16m0.181732815s) [upload=55545]: Next returned nil segment Aug 10 22:42:05.750671300 (53h16m0.181751431s) [upload=55545]: Waiting for error group managing segments... Aug 10 22:42:06.068503390 (53h16m0.499583521s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.069092899 (53h16m0.50017303s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.107858230 (53h16m0.538938361s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.120952784 (53h16m0.552032915s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.344626386 (53h16m0.775706517s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.472396183 (53h16m0.903476314s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.599585118 (53h16m1.030665249s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.602989492 (53h16m1.034069613s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.641577704 (53h16m1.072657835s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.660099663 (53h16m1.091179784s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.806051468 (53h16m1.237131599s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.835071855 (53h16m1.266151986s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.841828643 (53h16m1.272908764s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853893837 (53h16m1.284973968s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853915288 (53h16m1.284995419s) [seg_pos={0 0}, upload=55545]: Segment reached optimal threshold of 14 pieces. Aug 10 22:42:06.854012876 (53h16m1.285092997s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qhGin74NgTpgUFEFrMdhXaQC3v17PefWnKgLSotok56yRoxZx): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854066027 (53h16m1.285146158s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854148315 (53h16m1.285228446s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854157402 (53h16m1.285237533s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854194522 (53h16m1.285274654s) [seg_pos={0 0}, upload=55545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:42:06.854287050 (53h16m1.285367181s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:42:06.854291298 (53h16m1.285371429s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:42:07.702881346 (53h16m2.133961477s) [upload=55545]: Done uploading segments... Aug 10 22:42:07.702904620 (53h16m2.133984791s) [upload=55545]: Upload finished. err: <nil> Duration: 6.280943964s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X...............................X........................] duration: 1.83226327s FAIL (2025-08-10 22:49:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.832294481s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.380161721s FAIL (2025-08-10 22:49:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 6.380182561s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............X..X...............X.......X.X.....X.X.........] duration: 8.403560786s FAIL (2025-08-10 22:49:16 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 10 22:42:04.151608101 (53h15m58.582688232s) [upload=55545]: Starting upload Aug 10 22:42:04.151655141 (53h15m58.582735272s) [upload=55545]: Done starting upload Aug 10 22:42:04.151712230 (53h15m58.582792351s) [upload=55545]: Uploading segments... Aug 10 22:42:04.151832401 (53h15m58.582912532s) [upload=55545]: Got next segment. Inline: false Aug 10 22:42:04.151867177 (53h15m58.582947298s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:42:04.151871315 (53h15m58.582951446s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:42:04.596187241 (53h15m59.027267432s) [seg_pos={0 0}, upload=55545]: Begin upload segment... Aug 10 22:42:04.596256584 (53h15m59.027336715s) [seg_pos={0 0}, upload=55545]: Done begin upload segment. Aug 10 22:42:04.596302321 (53h15m59.027382462s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.596321418 (53h15m59.027401559s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597244444 (53h15m59.028324605s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597424017 (53h15m59.028504189s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597875963 (53h15m59.028956104s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598094571 (53h15m59.029174712s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598428159 (53h15m59.02950832s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598838364 (53h15m59.029918495s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599186390 (53h15m59.030266521s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599809602 (53h15m59.030889733s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599928190 (53h15m59.031008321s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600314469 (53h15m59.0313946s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600730284 (53h15m59.031810415s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.601867690 (53h15m59.032947821s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.602627183 (53h15m59.033707314s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605005474 (53h15m59.036085615s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605487957 (53h15m59.036568098s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605924182 (53h15m59.037004323s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:05.750652684 (53h16m0.181732815s) [upload=55545]: Next returned nil segment Aug 10 22:42:05.750671300 (53h16m0.181751431s) [upload=55545]: Waiting for error group managing segments... Aug 10 22:42:06.068503390 (53h16m0.499583521s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.069092899 (53h16m0.50017303s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.107858230 (53h16m0.538938361s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.120952784 (53h16m0.552032915s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.344626386 (53h16m0.775706517s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.472396183 (53h16m0.903476314s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.599585118 (53h16m1.030665249s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.602989492 (53h16m1.034069613s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.641577704 (53h16m1.072657835s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.660099663 (53h16m1.091179784s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.806051468 (53h16m1.237131599s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.835071855 (53h16m1.266151986s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.841828643 (53h16m1.272908764s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853893837 (53h16m1.284973968s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853915288 (53h16m1.284995419s) [seg_pos={0 0}, upload=55545]: Segment reached optimal threshold of 14 pieces. Aug 10 22:42:06.854012876 (53h16m1.285092997s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qhGin74NgTpgUFEFrMdhXaQC3v17PefWnKgLSotok56yRoxZx): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854066027 (53h16m1.285146158s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854148315 (53h16m1.285228446s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854157402 (53h16m1.285237533s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854194522 (53h16m1.285274654s) [seg_pos={0 0}, upload=55545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:42:06.854287050 (53h16m1.285367181s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:42:06.854291298 (53h16m1.285371429s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:42:07.702881346 (53h16m2.133961477s) [upload=55545]: Done uploading segments... Aug 10 22:42:07.702904620 (53h16m2.133984791s) [upload=55545]: Upload finished. err: <nil> Duration: 8.403587767s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 6.70407685s FAIL (2025-08-10 22:48:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.704103492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X...............................X.......................] duration: 1.878197294s FAIL (2025-08-10 22:48:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.878221362s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.188516714s FAIL (2025-08-10 22:48:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 6.188544688s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 6.646744358s FAIL (2025-08-10 22:48:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.64676658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............X..X...............X.......X.X.....X.X........] duration: 7.925714829s FAIL (2025-08-10 22:48:28 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 10 22:42:04.151608101 (53h15m58.582688232s) [upload=55545]: Starting upload Aug 10 22:42:04.151655141 (53h15m58.582735272s) [upload=55545]: Done starting upload Aug 10 22:42:04.151712230 (53h15m58.582792351s) [upload=55545]: Uploading segments... Aug 10 22:42:04.151832401 (53h15m58.582912532s) [upload=55545]: Got next segment. Inline: false Aug 10 22:42:04.151867177 (53h15m58.582947298s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:42:04.151871315 (53h15m58.582951446s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:42:04.596187241 (53h15m59.027267432s) [seg_pos={0 0}, upload=55545]: Begin upload segment... Aug 10 22:42:04.596256584 (53h15m59.027336715s) [seg_pos={0 0}, upload=55545]: Done begin upload segment. Aug 10 22:42:04.596302321 (53h15m59.027382462s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.596321418 (53h15m59.027401559s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597244444 (53h15m59.028324605s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597424017 (53h15m59.028504189s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597875963 (53h15m59.028956104s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598094571 (53h15m59.029174712s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598428159 (53h15m59.02950832s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598838364 (53h15m59.029918495s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599186390 (53h15m59.030266521s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599809602 (53h15m59.030889733s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599928190 (53h15m59.031008321s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600314469 (53h15m59.0313946s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600730284 (53h15m59.031810415s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.601867690 (53h15m59.032947821s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.602627183 (53h15m59.033707314s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605005474 (53h15m59.036085615s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605487957 (53h15m59.036568098s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605924182 (53h15m59.037004323s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:05.750652684 (53h16m0.181732815s) [upload=55545]: Next returned nil segment Aug 10 22:42:05.750671300 (53h16m0.181751431s) [upload=55545]: Waiting for error group managing segments... Aug 10 22:42:06.068503390 (53h16m0.499583521s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.069092899 (53h16m0.50017303s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.107858230 (53h16m0.538938361s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.120952784 (53h16m0.552032915s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.344626386 (53h16m0.775706517s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.472396183 (53h16m0.903476314s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.599585118 (53h16m1.030665249s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.602989492 (53h16m1.034069613s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.641577704 (53h16m1.072657835s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.660099663 (53h16m1.091179784s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.806051468 (53h16m1.237131599s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.835071855 (53h16m1.266151986s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.841828643 (53h16m1.272908764s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853893837 (53h16m1.284973968s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853915288 (53h16m1.284995419s) [seg_pos={0 0}, upload=55545]: Segment reached optimal threshold of 14 pieces. Aug 10 22:42:06.854012876 (53h16m1.285092997s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qhGin74NgTpgUFEFrMdhXaQC3v17PefWnKgLSotok56yRoxZx): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854066027 (53h16m1.285146158s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854148315 (53h16m1.285228446s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854157402 (53h16m1.285237533s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854194522 (53h16m1.285274654s) [seg_pos={0 0}, upload=55545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:42:06.854287050 (53h16m1.285367181s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:42:06.854291298 (53h16m1.285371429s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:42:07.702881346 (53h16m2.133961477s) [upload=55545]: Done uploading segments... Aug 10 22:42:07.702904620 (53h16m2.133984791s) [upload=55545]: Upload finished. err: <nil> Duration: 7.925742782s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X...............................X......................] duration: 1.903017325s FAIL (2025-08-10 22:48:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.903049798s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 7.585026434s FAIL (2025-08-10 22:48:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 7.585047044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 8.324978246s FAIL (2025-08-10 22:47:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 8.325009136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............X..X...............X.......X.X.....X.X.......] duration: 8.408562743s FAIL (2025-08-10 22:47:46 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 10 22:42:04.151608101 (53h15m58.582688232s) [upload=55545]: Starting upload Aug 10 22:42:04.151655141 (53h15m58.582735272s) [upload=55545]: Done starting upload Aug 10 22:42:04.151712230 (53h15m58.582792351s) [upload=55545]: Uploading segments... Aug 10 22:42:04.151832401 (53h15m58.582912532s) [upload=55545]: Got next segment. Inline: false Aug 10 22:42:04.151867177 (53h15m58.582947298s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:42:04.151871315 (53h15m58.582951446s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:42:04.596187241 (53h15m59.027267432s) [seg_pos={0 0}, upload=55545]: Begin upload segment... Aug 10 22:42:04.596256584 (53h15m59.027336715s) [seg_pos={0 0}, upload=55545]: Done begin upload segment. Aug 10 22:42:04.596302321 (53h15m59.027382462s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.596321418 (53h15m59.027401559s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597244444 (53h15m59.028324605s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597424017 (53h15m59.028504189s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597875963 (53h15m59.028956104s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598094571 (53h15m59.029174712s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598428159 (53h15m59.02950832s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598838364 (53h15m59.029918495s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599186390 (53h15m59.030266521s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599809602 (53h15m59.030889733s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599928190 (53h15m59.031008321s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600314469 (53h15m59.0313946s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600730284 (53h15m59.031810415s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.601867690 (53h15m59.032947821s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.602627183 (53h15m59.033707314s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605005474 (53h15m59.036085615s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605487957 (53h15m59.036568098s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605924182 (53h15m59.037004323s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:05.750652684 (53h16m0.181732815s) [upload=55545]: Next returned nil segment Aug 10 22:42:05.750671300 (53h16m0.181751431s) [upload=55545]: Waiting for error group managing segments... Aug 10 22:42:06.068503390 (53h16m0.499583521s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.069092899 (53h16m0.50017303s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.107858230 (53h16m0.538938361s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.120952784 (53h16m0.552032915s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.344626386 (53h16m0.775706517s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.472396183 (53h16m0.903476314s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.599585118 (53h16m1.030665249s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.602989492 (53h16m1.034069613s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.641577704 (53h16m1.072657835s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.660099663 (53h16m1.091179784s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.806051468 (53h16m1.237131599s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.835071855 (53h16m1.266151986s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.841828643 (53h16m1.272908764s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853893837 (53h16m1.284973968s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853915288 (53h16m1.284995419s) [seg_pos={0 0}, upload=55545]: Segment reached optimal threshold of 14 pieces. Aug 10 22:42:06.854012876 (53h16m1.285092997s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qhGin74NgTpgUFEFrMdhXaQC3v17PefWnKgLSotok56yRoxZx): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854066027 (53h16m1.285146158s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854148315 (53h16m1.285228446s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854157402 (53h16m1.285237533s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854194522 (53h16m1.285274654s) [seg_pos={0 0}, upload=55545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:42:06.854287050 (53h16m1.285367181s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:42:06.854291298 (53h16m1.285371429s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:42:07.702881346 (53h16m2.133961477s) [upload=55545]: Done uploading segments... Aug 10 22:42:07.702904620 (53h16m2.133984791s) [upload=55545]: Upload finished. err: <nil> Duration: 8.408591096s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X...............................X.....................] duration: 1.98746558s FAIL (2025-08-10 22:47:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.987497281s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [................X..X...............X.......X.X.....X.X......] duration: 7.72027555s FAIL (2025-08-10 22:47:09 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 10 22:42:04.151608101 (53h15m58.582688232s) [upload=55545]: Starting upload Aug 10 22:42:04.151655141 (53h15m58.582735272s) [upload=55545]: Done starting upload Aug 10 22:42:04.151712230 (53h15m58.582792351s) [upload=55545]: Uploading segments... Aug 10 22:42:04.151832401 (53h15m58.582912532s) [upload=55545]: Got next segment. Inline: false Aug 10 22:42:04.151867177 (53h15m58.582947298s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:42:04.151871315 (53h15m58.582951446s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 10 22:42:04.596187241 (53h15m59.027267432s) [seg_pos={0 0}, upload=55545]: Begin upload segment... Aug 10 22:42:04.596256584 (53h15m59.027336715s) [seg_pos={0 0}, upload=55545]: Done begin upload segment. Aug 10 22:42:04.596302321 (53h15m59.027382462s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.596321418 (53h15m59.027401559s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597244444 (53h15m59.028324605s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597424017 (53h15m59.028504189s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.597875963 (53h15m59.028956104s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598094571 (53h15m59.029174712s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598428159 (53h15m59.02950832s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.598838364 (53h15m59.029918495s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599186390 (53h15m59.030266521s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599809602 (53h15m59.030889733s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.599928190 (53h15m59.031008321s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600314469 (53h15m59.0313946s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.600730284 (53h15m59.031810415s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.601867690 (53h15m59.032947821s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.602627183 (53h15m59.033707314s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605005474 (53h15m59.036085615s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605487957 (53h15m59.036568098s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:04.605924182 (53h15m59.037004323s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Uploading piece... Aug 10 22:42:05.750652684 (53h16m0.181732815s) [upload=55545]: Next returned nil segment Aug 10 22:42:05.750671300 (53h16m0.181751431s) [upload=55545]: Waiting for error group managing segments... Aug 10 22:42:06.068503390 (53h16m0.499583521s) [noise=true , address=193.149.164.143:4700 , piece_id=G6KEA2ZLTO3YHKWAMP7K3JM4G4AXCOTTSGJVRYO3DYFVPN2Y4UXA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.069092899 (53h16m0.50017303s) [noise=true , address=64.49.13.139:4470 , piece_id=7SZXL5JQNRCIGGFHOXVZWLNRDEEAR5QTN4IHHUS4NZU543TRC6CQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.107858230 (53h16m0.538938361s) [noise=true , address=64.49.13.26:4956 , piece_id=5UXTJFRSGZIHZSM6L2ATCSPSCY5J3F3RVUDGI45JQRH3OZSWJKGQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.120952784 (53h16m0.552032915s) [noise=true , address=64.49.13.138:4492 , piece_id=J3ZOQMMZDCHI4K2FPR7NJYDUUBBH5FOOIJFUHI6RZ7TQXDSLVAEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.344626386 (53h16m0.775706517s) [noise=true , address=64.49.13.155:4916 , piece_id=2AOWUVE2QIO366DL4ZVOGBUVFYLLFIVPQQ6PKAZ4ZTLQVW5MCVXQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.472396183 (53h16m0.903476314s) [noise=true , address=79.127.219.47:4048 , piece_id=ZACE7EELSHBD3XRHYBX7AGNRFTFQBARYMTRJBUIZYRI4BMT75JAQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.599585118 (53h16m1.030665249s) [noise=true , address=104.36.82.35:4586 , piece_id=YWIKOYWKQY4CQE3W7IKKAF3LR4EPS4H5KOJRTDMQA2S6LSQWZXZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.602989492 (53h16m1.034069613s) [noise=true , address=104.36.82.59:4333 , piece_id=3V5N2ROKSTUHDMU7BYV56LYQYPHGJHP75RKTX2BVW4G5H5AMARIQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.641577704 (53h16m1.072657835s) [noise=true , address=104.36.82.14:4232 , piece_id=4AACJUGZPWEJ7ZVFPAKZT36DMBTLCOZY2HRIQHOOO4ZWNOJT3DHA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.660099663 (53h16m1.091179784s) [noise=true , address=104.36.82.11:4136 , piece_id=4MCGEVW7U54OAC7UWZJNCJI5IJUOHBWFMSAS4KDRCTU6VSWOW3DQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.806051468 (53h16m1.237131599s) [noise=true , address=79.127.201.215:4351 , piece_id=PUIKGZDMXFINOQDPNAJES6FFLX6SOZE44ENM5A6SGGV25QGJIW4A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.835071855 (53h16m1.266151986s) [noise=true , address=104.36.82.5:4853 , piece_id=HJEJQXGFUQZYN5UOCJQH2ZVTKCWMGJ7POCKB5SDHCVVAUCNHPAZA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.841828643 (53h16m1.272908764s) [noise=true , address=104.36.82.17:4351 , piece_id=E7WZ4HBNFNYJ62WQCMGII6VFOCYKH6A6EFVQRDSVVWR677HIQWEQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853893837 (53h16m1.284973968s) [noise=true , address=104.36.82.60:4341 , piece_id=2ZHQ4QAZEJMN5XWNJNULJZWH6WB4BXFGAHJOOIJUHW75IENX6HPA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: <nil> Aug 10 22:42:06.853915288 (53h16m1.284995419s) [seg_pos={0 0}, upload=55545]: Segment reached optimal threshold of 14 pieces. Aug 10 22:42:06.854012876 (53h16m1.285092997s) [noise=true , address=104.36.82.8:4991 , piece_id=423DVBEHU2LE3KSK2LKQ6JLXP23EJ5TE4AFFJT4G74JEUDJTTINA, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qhGin74NgTpgUFEFrMdhXaQC3v17PefWnKgLSotok56yRoxZx): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854066027 (53h16m1.285146158s) [noise=true , address=104.36.82.47:4540 , piece_id=2WY5A74WYGG35WL675HOWPND7T7OER2OJOWKEDW5YXF577WB3F5A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854148315 (53h16m1.285228446s) [noise=true , address=104.36.82.23:4455 , piece_id=ENPE5ECCA5G3FCSNBQ5LHUOQ5DVJ46OB3IV4QEJIAPLUYHKNC2AQ, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854157402 (53h16m1.285237533s) [noise=true , address=104.36.82.38:4706 , piece_id=VN4OJLLEXPPO7ASAQ6CGIEK54YLUP33VLBXSWNGQBKHHNBMH265A, seg_pos={0 0}, upload=55545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV): protocol: expected piece hash; context canceled; EOF Aug 10 22:42:06.854194522 (53h16m1.285274654s) [seg_pos={0 0}, upload=55545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 10 22:42:06.854287050 (53h16m1.285367181s) [upload=55545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 10 22:42:06.854291298 (53h16m1.285371429s) [upload=55545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:42:07.702881346 (53h16m2.133961477s) [upload=55545]: Done uploading segments... Aug 10 22:42:07.702904620 (53h16m2.133984791s) [upload=55545]: Upload finished. err: <nil> Duration: 7.720300569s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 7.354018267s FAIL (2025-08-10 22:47:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== loading piece... Aug 10 22:40:53.943549380 (53h14m48.374629521s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943556974 (53h14m48.374637095s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943597543 (53h14m48.374677674s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.943973151 (53h14m48.375053282s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944299355 (53h14m48.375379496s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944384608 (53h14m48.375464739s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944631009 (53h14m48.37571114s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944727565 (53h14m48.375807696s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.944860439 (53h14m48.37594057s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945088966 (53h14m48.376169097s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945147738 (53h14m48.376227869s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945353152 (53h14m48.376433283s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945606616 (53h14m48.376686748s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.945871724 (53h14m48.376951855s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.947941243 (53h14m48.379021374s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:53.948327823 (53h14m48.379407954s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Uploading piece... Aug 10 22:40:55.065322831 (53h14m49.496402962s) [upload=55525]: Next returned nil segment Aug 10 22:40:55.065342348 (53h14m49.49642251s) [upload=55525]: Waiting for error group managing segments... Aug 10 22:40:55.351936440 (53h14m49.783016571s) [noise=true , address=64.49.13.138:4488 , piece_id=RCCTYN7Y2OZIYWEKPURPYO4BSOLDHVHTCO63Y3BH5N46CN7CNXIA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.396100421 (53h14m49.827180552s) [noise=true , address=64.49.13.139:4470 , piece_id=DLZLLGII6NXMCWTFWGS3OF7LPULMLR4JPQMDKK6GCUN2DAMX625A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.406362685 (53h14m49.837442826s) [noise=true , address=193.149.164.143:4701 , piece_id=QHI2XVOX7W2UGF4BQSLSD2VIVV52LLABIQFT3BXLZEUNG3ZLPIEA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.419436730 (53h14m49.850516861s) [noise=true , address=79.127.205.245:4240 , piece_id=XF4MLNG6VPY2WDQBQHEKZQA64TDPTL6WSAI6ZW6XWPEQC4ODBPWA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:40:55.430290006 (53h14m49.861370137s) [noise=true , address=79.127.205.234:4598 , piece_id=2EI6HK3UTDZ5XOY34OVQG3EKCD7SFPICWTSJHH4DQJBFOSETL5RQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:17.485741365 (53h15m11.916821486s) [noise=true , address=104.36.82.46:4498 , piece_id=BOR2M6TCV2A5NHT4X62SF66ZMQUE2YB2WBNH56ANZTE367RACQXA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:24.084063242 (53h15m18.515143373s) [noise=true , address=104.36.82.23:4452 , piece_id=P4PEW5G2HXNZSASQCYCWYTQFYI6XG63A5INLBHVOOCE5HH4BPRHQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:28.527843834 (53h15m22.958923965s) [noise=true , address=104.36.82.47:4541 , piece_id=LC5QTRQGGXQ6CG4VKGGLKPFLBPOZ46I5IUN5EEVRZNBXCLIV4CCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:33.874254204 (53h15m28.305334335s) [noise=true , address=104.36.82.37:4677 , piece_id=7FGAV3NNV7GBZK5MNEI35SXTGL2JZ7FHKYOHFSSFPAOCZJ5UI75Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:47.512694556 (53h15m41.943774687s) [noise=true , address=104.36.82.38:4722 , piece_id=SMBUNZENTKSSKSSYLKBRD6QA4FVMYFA655OBQXNW4FWYUQELTNCA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: <nil> Aug 10 22:41:52.846710159 (53h15m47.27779029s) [noise=true , address=104.36.82.45:4911 , piece_id=S4CVGLLIK3C62VB4WQKD6SER3AL3RVG235HXO2DD7VSFFKVI6IRQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12mzM4hseStJYeWqqm29qGjdaKkxydqFsKaKwNFrywGpDWSwFDQ, address:104.36.82.45:4911): protocol: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60468->104.36.82.45:4911: use of closed network connection Aug 10 22:41:52.846755196 (53h15m47.277835327s) [noise=true , address=104.36.82.35:4612 , piece_id=HIICHFXV4DN3VWYY4VS6CGRPDV2XIX25DHLJVSR3PIPUSPQRCWVA, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:104.36.82.35:4612): protocol: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->104.36.82.35:4612: use of closed network connection Aug 10 22:41:52.846783941 (53h15m47.277864072s) [noise=true , address=104.36.82.60:4339 , piece_id=4TQXIBW4F65FX25ZHDGVYO7ZJIZKJYZR6M5F32L6MBLMGAUVQIOQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846820601 (53h15m47.277900722s) [noise=true , address=104.36.82.8:4977 , piece_id=GLYGCEHBCWPC2UGZKCPXVOM2BI34I2DL552Z7YZCLZ5YODKDIXBQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT, address:104.36.82.8:4977): protocol: rpc: dial tcp 104.36.82.8:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846853675 (53h15m47.277933806s) [noise=true , address=104.36.82.58:4279 , piece_id=PF6K7Q7GMBN237YTKOCSKUCSIVG4WZHAB5VM7XNQ7SZEVVJY2D4A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846860007 (53h15m47.277940138s) [noise=true , address=104.36.82.62:4430 , piece_id=U63SQHNDHSNPB3Y3OC4AOSPEAIKD5TI6COFPP7TXPCCSSIQM3WKQ, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:15fmq7zLwrSHRmwJ4eai2bXDnCsAqUEhqCLHqHvMuf189v9sQb, address:104.36.82.62:4430): protocol: write tcp 188.166.225.170:53320->104.36.82.62:4430: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846875696 (53h15m47.277955817s) [noise=true , address=104.36.82.10:4104 , piece_id=XA3M5X4ZYV75QK5M3OQL55HDTF6CYS5ZNOMUC5HNH3HNXSZDPF7Q, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:12th1zHy3xbqtrsUMY78iYg8WhCoubrbsuQXtLBcAC8hjEfo1NH, address:104.36.82.10:4104): protocol: rpc: dial tcp 104.36.82.10:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846892138 (53h15m47.277972269s) [noise=true , address=104.36.82.18:4372 , piece_id=23XSYB4TPDQFFLVPRHHDOCA4AOAYJMQZMFPDT7EQHWHMPU6KT23A, seg_pos={0 0}, upload=55525]: Done uploading piece. err: ecclient: upload failed (node:1fchsL2RkNzQhjmtJgKigZUHhrVgb48PohuJNFk8GmKiVax44T, address:104.36.82.18:4372): protocol: rpc: dial tcp 104.36.82.18:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:41:52.846913489 (53h15m47.27799362s) [seg_pos={0 0}, upload=55525]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 10 22:41:53.081503289 (53h15m47.51258341s) [upload=55525]: Done uploading segments... Aug 10 22:41:53.081530240 (53h15m47.512610362s) [upload=55525]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 7.354044398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 8.036162611s FAIL (2025-08-10 22:47:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 13 but got 0) ======LOGS====== ading piece... Aug 10 22:39:50.336074364 (53h13m44.767154495s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Uploading piece... Aug 10 22:40:19.456819087 (53h14m13.887899229s) [upload=55509]: Next segment err: context deadline exceeded Aug 10 22:40:19.457783993 (53h14m13.888864135s) [noise=true , address=45.91.116.5:4298 , piece_id=GVMGOZCJWYS7J3T6DHGZE2FV6HL765AH2GKAUA6YC4WO5VNYHK5Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38014->45.91.116.5:4298: use of closed network connection Aug 10 22:40:19.461082815 (53h14m13.892162946s) [noise=true , address=45.91.116.11:4119 , piece_id=GKGBSCCL2OE2QOW64DNOYLI6B3DRY42DRZFSYPHQPW4ZZJI3WZMQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34580->45.91.116.11:4119: use of closed network connection Aug 10 22:40:19.461163860 (53h14m13.892243991s) [noise=true , address=77.90.31.10:4519 , piece_id=54QOA6EMWRSZQ6Y2OJLISSTZZHXQ65BMVTPKSFSXTN7FX6GYSP4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53246->77.90.31.10:4519: use of closed network connection Aug 10 22:40:19.461232402 (53h14m13.892312533s) [noise=true , address=104.36.82.14:4222 , piece_id=YTNRW3RFBMSRNON7BINBR6CQ4RJ4RAUB22SRHKICXJHSB2RBTANQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461288870 (53h14m13.892369011s) [noise=true , address=104.36.82.8:4970 , piece_id=MK6SEEA5GOO633T5M3PZEBAI3SLIXHBJ6K6OC3VS26AUNF3BODCQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 10 22:40:19.461349325 (53h14m13.892429466s) [noise=true , address=45.91.116.14:4190 , piece_id=WV3CT6AZVKYWCVFRX4A64OWDA3MDARBNZXZVAUQEPITZDHEJM4NQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41494->45.91.116.14:4190: use of closed network connection Aug 10 22:40:19.461405783 (53h14m13.892485914s) [noise=true , address=77.90.31.13:4627 , piece_id=FDUYEH66CG5N75FFPZO33VLDNXERXVRU777HRENJQ6AO3CE3PGXA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60098->77.90.31.13:4627: use of closed network connection Aug 10 22:40:19.461459857 (53h14m13.892539988s) [noise=true , address=45.91.116.4:4940 , piece_id=EGD74UMVIBGMBYFTATXD5MP736KKHKG3S4Z6XM6YIVRBQMS7WNFA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42802->45.91.116.4:4940: use of closed network connection Aug 10 22:40:19.461515543 (53h14m13.892595674s) [noise=true , address=77.90.31.17:4770 , piece_id=UCGDKAZLEO5NCHXBXPV6WS45GX4DQ6XMZV5OVOTH3YPZ76IGKMTQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43000->77.90.31.17:4770: use of closed network connection Aug 10 22:40:19.461591038 (53h14m13.892671169s) [noise=true , address=45.91.116.12:4225 , piece_id=EG2YTLPB7OYMJALBKBYNX4Y7U4WZO35MYSMRYI7YW3EQHYPTCLVA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->45.91.116.12:4225: use of closed network connection Aug 10 22:40:19.461652836 (53h14m13.892732967s) [noise=true , address=77.90.31.19:4841 , piece_id=FAYN4ZUGPDD2PLZOF3Z75UMYDIKQVWBFLDZ66IMEJS3PGZHLIN6A, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33564->77.90.31.19:4841: use of closed network connection Aug 10 22:40:19.461732308 (53h14m13.892812449s) [noise=true , address=77.90.31.12:4592 , piece_id=I47565EJHMGJPCSUXVXEWRZDHDSAJJAGCPCO7XTPLTIGY3IQW2ZQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42676->77.90.31.12:4592: use of closed network connection Aug 10 22:40:19.461887956 (53h14m13.892968087s) [noise=true , address=77.90.31.12:4590 , piece_id=XMSOJVMRJHDL7LL7FEJIBMQWXYN6J52B6GBLV6LV4XZIDHR47XYA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58782->77.90.31.12:4590: use of closed network connection Aug 10 22:40:19.461959753 (53h14m13.893039874s) [noise=true , address=45.91.116.10:4083 , piece_id=X4JN6C2QK55BWAUI7R4PMROM5VI5K3IZQBVLYLEWKVPDBQZHKB4Q, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46198->45.91.116.10:4083: use of closed network connection Aug 10 22:40:19.462013736 (53h14m13.893093867s) [noise=true , address=77.90.31.11:4554 , piece_id=UNJ3KNUZDBS7FW67TI7FHKVOVX4SQCLT5NEKRPOM4JS3BJI73CNQ, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->77.90.31.11:4554: use of closed network connection Aug 10 22:40:19.462072048 (53h14m13.893152179s) [noise=true , address=45.91.116.7:4009 , piece_id=6XPXA2QEPR7YLCVC3VED6BO5YMJ3QDZQ66QO5O3ZQUTOPAF7I6AA, seg_pos={0 0}, upload=55509]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->45.91.116.7:4009: use of closed network connection Aug 10 22:40:19.462087708 (53h14m13.893167839s) [seg_pos={0 0}, upload=55509]: Done waiting for segment. successful: 0 optimal: 13 errs: <nil> Duration: 8.036184874s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X...............................X....................] duration: 1.906372124s FAIL (2025-08-10 22:47:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 10 22:33:23.090759975 (53h7m17.521840126s) [upload=55388]: Starting upload Aug 10 22:33:23.090831261 (53h7m17.521911422s) [upload=55388]: Done starting upload Aug 10 22:33:23.090903269 (53h7m17.52198342s) [upload=55388]: Uploading segments... Aug 10 22:33:23.090955248 (53h7m17.52203541s) [upload=55388]: Got next segment. Inline: true Aug 10 22:33:23.090961811 (53h7m17.522041962s) [upload=55388]: Waiting for error group managing segments... Aug 10 22:33:23.091016636 (53h7m17.522096807s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 10 22:33:23.091023408 (53h7m17.52210356s) [upload=55388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 10 22:33:23.091028138 (53h7m17.522108289s) [upload=55388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 10 22:33:23.358287015 (53h7m17.789367146s) [upload=55388]: Done uploading segments... Aug 10 22:33:23.358303086 (53h7m17.789383207s) [upload=55388]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.906393196s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)