Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 32 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X...........................] duration: 2.386287051s FAIL (2025-08-07 22:00:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.3863022s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 29 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXX..XX..X.................] duration: 5.974481311s FAIL (2025-08-07 22:00:46 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.974505878s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 26 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXX.....X..................] duration: 6.840277594s FAIL (2025-08-07 22:00:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== t segment. Inline: false Aug 7 21:44:25.061589945 (12h4m24.650934922s) [upload=12552]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:25.061594724 (12h4m24.650939711s) [upload=12552]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:44:50.602885060 (12h4m50.192230047s) [seg_pos={0 0}, upload=12552]: Begin upload segment... Aug 7 21:44:50.602935446 (12h4m50.192280423s) [seg_pos={0 0}, upload=12552]: Done begin upload segment. Aug 7 21:44:50.603112565 (12h4m50.192457552s) [noise=true , address=79.127.205.233:4624 , piece_id=WEAYHGF7VE6W5ML3QAE3AITMEJGQFH4JQZ7RQKR7O4NKTPR6RZRA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.603695610 (12h4m50.193040597s) [noise=true , address=104.36.82.22:4529 , piece_id=ZTTBM5H6MHVMQ3OFVTVISN6WJ7WB3XBWI7YZKUXCFNQ6NO3D4CGA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.604300768 (12h4m50.193645755s) [noise=true , address=45.91.116.6:4002 , piece_id=AE5EL7NWD6FEQ7NXX2JRHB4MVZQDXRGQU3EER2X7COISFAX6P3XQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.604588519 (12h4m50.193933496s) [noise=true , address=104.36.82.56:4249 , piece_id=RD4B7IDIJV25L7EJNSYCZGW5QKJEM435E2N3UFUD44Q4S3DCBOXQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605153860 (12h4m50.194498837s) [noise=true , address=77.90.31.11:4562 , piece_id=XC6F2WLSXKQB7E7MBRNZBJKUSUFL6XNA5TVX5M3ZJI3ZZMK7UBLA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605455377 (12h4m50.194800364s) [noise=true , address=104.36.82.23:4466 , piece_id=5J5X2LSP6GCX2PNRIDZYFJPISWSX3YSRID5M2WU6Y5Z6LWRHLC4Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605743810 (12h4m50.195088797s) [noise=true , address=104.36.82.62:4420 , piece_id=UOO36PYVE5WFRRZLGYLJ6I6SV4EO7W2ZH2TIXJ2OIZZMY3LTHR2A, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.606547457 (12h4m50.195892444s) [noise=true , address=104.36.82.5:4842 , piece_id=E5DSJXN63RFX7EQ4ZTBMUESFW77EB3HQNA5KCQGRVRAA5XO2GN3Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.606982099 (12h4m50.196327086s) [noise=true , address=104.36.82.59:4301 , piece_id=VVR5CQWCPNPZUFEFCPB6ZL2KASWS5XO65FVDI474RKRZSJFDUQTA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.607295849 (12h4m50.196640836s) [noise=true , address=45.91.116.14:4198 , piece_id=2HRU3JZDQSNGD5NJZDLZSRG2WIO2Y2QBSOAY6KUEGUKDBWM53RSA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.607695593 (12h4m50.19704058s) [noise=true , address=104.36.82.6:4881 , piece_id=VUC33D7TLVFICDLA3334DEKBJE5E5P4HQTJU3VDQNMVYWVESSJWA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.608364103 (12h4m50.19770909s) [noise=true , address=104.36.82.2:4757 , piece_id=H6KNYP5HASPWQIHADTMEZDC5AE5HITCAY73XGNWVEXJOPDEZ3SHA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.608764288 (12h4m50.198109265s) [noise=true , address=104.36.82.46:4493 , piece_id=PS4S4OZ6HSITEJFPJDIBWQ5RGWTDRKMX3NCEWXCLC2PSP5Q2KPZQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609065284 (12h4m50.198410271s) [noise=true , address=104.36.82.47:4518 , piece_id=KJ3PJCT3EDH3WZBT7AWWKI6ABGVVAMKZTVJY7Q76FULC72TUWA6Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609312777 (12h4m50.198657755s) [noise=true , address=104.36.82.60:4349 , piece_id=GTTF32YMJWBY2GZEFZPIBZXZWNI7MBLUYCTQ3SRFG735PLWXTXBQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609589888 (12h4m50.198934875s) [noise=true , address=104.36.82.52:4116 , piece_id=7QA2Q5ACN43TQUOKWA34EZFSSQBE3OWNJTYODHHQ5GCVOA4QZOJA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.610295969 (12h4m50.199641056s) [noise=true , address=104.36.82.37:4678 , piece_id=VJO5JO3SJEXBQZC6PJ34DRKWEQKOPI33ZF4IHNNOOTSD7IBRR4PQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.610811305 (12h4m50.200156302s) [noise=true , address=64.49.13.169:4714 , piece_id=A6O3H7QLE6KQTHWUKOEROVHPEHVLG22KNF2KCV3HK2KCLXP7SYVA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:51.715421583 (12h4m51.30476657s) [upload=12552]: Next returned nil segment Aug 7 21:44:51.715438125 (12h4m51.304783112s) [upload=12552]: Waiting for error group managing segments... Aug 7 21:44:51.998131281 (12h4m51.587476268s) [noise=true , address=64.49.13.169:4714 , piece_id=A6O3H7QLE6KQTHWUKOEROVHPEHVLG22KNF2KCV3HK2KCLXP7SYVA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:51.999236335 (12h4m51.588581312s) [noise=true , address=79.127.205.233:4624 , piece_id=WEAYHGF7VE6W5ML3QAE3AITMEJGQFH4JQZ7RQKR7O4NKTPR6RZRA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.638299991 (12h4m52.227644978s) [noise=true , address=45.91.116.6:4002 , piece_id=AE5EL7NWD6FEQ7NXX2JRHB4MVZQDXRGQU3EER2X7COISFAX6P3XQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.639927565 (12h4m52.229272542s) [noise=true , address=104.36.82.6:4881 , piece_id=VUC33D7TLVFICDLA3334DEKBJE5E5P4HQTJU3VDQNMVYWVESSJWA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.643220585 (12h4m52.232565572s) [noise=true , address=104.36.82.2:4757 , piece_id=H6KNYP5HASPWQIHADTMEZDC5AE5HITCAY73XGNWVEXJOPDEZ3SHA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.711872926 (12h4m52.301217913s) [noise=true , address=104.36.82.56:4249 , piece_id=RD4B7IDIJV25L7EJNSYCZGW5QKJEM435E2N3UFUD44Q4S3DCBOXQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.763601896 (12h4m52.352946883s) [noise=true , address=104.36.82.22:4529 , piece_id=ZTTBM5H6MHVMQ3OFVTVISN6WJ7WB3XBWI7YZKUXCFNQ6NO3D4CGA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.768238797 (12h4m52.357583785s) [noise=true , address=104.36.82.60:4349 , piece_id=GTTF32YMJWBY2GZEFZPIBZXZWNI7MBLUYCTQ3SRFG735PLWXTXBQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.798543408 (12h4m52.387888395s) [noise=true , address=45.91.116.14:4198 , piece_id=2HRU3JZDQSNGD5NJZDLZSRG2WIO2Y2QBSOAY6KUEGUKDBWM53RSA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.861192704 (12h4m52.450537691s) [noise=true , address=77.90.31.11:4562 , piece_id=XC6F2WLSXKQB7E7MBRNZBJKUSUFL6XNA5TVX5M3ZJI3ZZMK7UBLA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:53.070218164 (12h4m52.659563151s) [noise=true , address=104.36.82.59:4301 , piece_id=VVR5CQWCPNPZUFEFCPB6ZL2KASWS5XO65FVDI474RKRZSJFDUQTA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:53.558930346 (12h4m53.148275333s) [noise=true , address=104.36.82.46:4493 , piece_id=PS4S4OZ6HSITEJFPJDIBWQ5RGWTDRKMX3NCEWXCLC2PSP5Q2KPZQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.132342777 (12h4m54.721687765s) [noise=true , address=104.36.82.37:4678 , piece_id=VJO5JO3SJEXBQZC6PJ34DRKWEQKOPI33ZF4IHNNOOTSD7IBRR4PQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.265236558 (12h4m54.854581545s) [noise=true , address=104.36.82.52:4116 , piece_id=7QA2Q5ACN43TQUOKWA34EZFSSQBE3OWNJTYODHHQ5GCVOA4QZOJA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.265254161 (12h4m54.854599138s) [seg_pos={0 0}, upload=12552]: Segment reached optimal threshold of 14 pieces. Aug 7 21:44:55.265482308 (12h4m54.854827295s) [noise=true , address=104.36.82.5:4842 , piece_id=E5DSJXN63RFX7EQ4ZTBMUESFW77EB3HQNA5KCQGRVRAA5XO2GN3Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn): protocol: expected piece hash; write tcp 188.166.225.170:57020->104.36.82.5:4842: use of closed network connection; write tcp 188.166.225.170:57020->104.36.82.5:4842: use of closed network connection; EOF Aug 7 21:44:55.265551430 (12h4m54.854896417s) [noise=true , address=104.36.82.47:4518 , piece_id=KJ3PJCT3EDH3WZBT7AWWKI6ABGVVAMKZTVJY7Q76FULC72TUWA6Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: expected piece hash; context canceled; EOF Aug 7 21:44:55.265766391 (12h4m54.855111378s) [noise=true , address=104.36.82.23:4466 , piece_id=5J5X2LSP6GCX2PNRIDZYFJPISWSX3YSRID5M2WU6Y5Z6LWRHLC4Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rkNnk9RUhKBJ3EsakqMRb3wYwrhRLhB4ZYbrdfKPDk8erRJpP): protocol: write tcp 188.166.225.170:49724->104.36.82.23:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:44:55.265824172 (12h4m54.855169159s) [noise=true , address=104.36.82.62:4420 , piece_id=UOO36PYVE5WFRRZLGYLJ6I6SV4EO7W2ZH2TIXJ2OIZZMY3LTHR2A, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SWVQwL5PxDLaSXADwTrAYS2hVcNBuYAA6ReWuRM3UZa57QyYi): protocol: read tcp 188.166.225.170:33092->104.36.82.62:4420: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:44:55.265833430 (12h4m54.855178417s) [seg_pos={0 0}, upload=12552]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:44:55.265888796 (12h4m54.855233783s) [upload=12552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:44:55.265892303 (12h4m54.85523728s) [upload=12552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:46.088275738 (12h8m45.677620725s) [upload=12552]: Done uploading segments... Aug 7 21:48:46.088301307 (12h8m45.677646284s) [upload=12552]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.840299918s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X......X..X.............X..X....................X........] duration: 7.078665395s FAIL (2025-08-07 22:00:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 57 (12h15m32.758009944s) [noise=true , address=104.36.82.54:4166 , piece_id=5SBUSGFVYPNFGILW6OTNLGJQG7CNI7GYYEJQJMITPKFZ2MAK5QRQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.168945643 (12h15m32.75829063s) [noise=true , address=104.36.82.12:4179 , piece_id=KFVQX32X6I7QBU552L4NSPZG3JTVT24YESONPWGM6UCTR4ZWBUKQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.168966093 (12h15m32.75831109s) [noise=true , address=104.36.82.43:4850 , piece_id=JH2ASD4NQAQMOBOZMDGFKVPPOS2P26QIFSYDNHYZN24RP52E4XLA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169201393 (12h15m32.75854638s) [noise=true , address=104.36.82.36:4633 , piece_id=6RL3INCLYJCEH53JOFJDL2MMEXGQ2G4T33PKZPJ3W2FG5QMCFLCQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169453466 (12h15m32.758798453s) [noise=true , address=104.36.82.47:4520 , piece_id=K75KQHBYH7QTZDCFM7ZW55EUO2W3UN2NEIKUMNCMQWHYYNOWLL3A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169588975 (12h15m32.758933962s) [noise=true , address=104.36.82.9:4014 , piece_id=D3NLTEDBRB4M7GIMPYCDNQ66HYLAZGSSBRM7SJWU45LPTCRMYMCQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170078472 (12h15m32.759423459s) [noise=true , address=104.36.82.39:4739 , piece_id=QGKQ3NJO7OTN5O2QQU43KYJXARJW6JQMSCONEJR35AEBBNRORJCA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170313160 (12h15m32.759658147s) [noise=true , address=104.36.82.62:4429 , piece_id=RCPKZHWWS56QSEVG6L25B3FMQOU5ZES3BNAEYIUYQ3PLQV3FDMVQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170515587 (12h15m32.759860574s) [noise=true , address=79.127.201.215:4353 , piece_id=II3JPNVAATS6APC4JGPPOJRHQRMHS5EYGLLEEAMTOPHPEGRQUF7A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170753382 (12h15m32.760098359s) [noise=true , address=45.91.116.8:4065 , piece_id=OEVAVKCHFYMJMGBKUKF5F34O3NH47DG45QJHHNWMOD7TDRMCQLAQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170965789 (12h15m32.760310776s) [noise=true , address=77.90.31.14:4678 , piece_id=7WZVWEAFKI4ZEIHTZOJNMAQWLBVPPLB7U4JNYZQNXCVRC2S4BUBA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171167525 (12h15m32.760512512s) [noise=true , address=104.36.82.45:4926 , piece_id=S3EH75AGZ3QVGR6RM3RI3SCHXZJPE7LORX2QQA5SCEADV3RDYM6A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171354372 (12h15m32.760699359s) [noise=true , address=64.49.13.154:4935 , piece_id=O6TZCBWBHVESLRX76EFTUT4U4OIKFFVXO5UKLMWAJA2BDTHIHN2A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171604611 (12h15m32.760949598s) [noise=true , address=104.36.82.44:4894 , piece_id=WCLKE2VTORDUBR5II5IQ5OTYMNHCMP2OYAIBHG3GV5P5U3FCO7HQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171788884 (12h15m32.761133861s) [noise=true , address=104.36.82.37:4674 , piece_id=AN3RCVXXXNEJUD33D4W47FDFYJQ7X5ZK7H4BJD6NR4PJOJSCVV6A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.181382449 (12h15m32.770727426s) [noise=true , address=104.36.82.23:4474 , piece_id=WM2SVCDZEFAN6VXQ2PWKBZGUCHIHNJ52RNKJKUNBMJ3AHFXUT3PA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:34.281100029 (12h15m33.870445006s) [upload=12734]: Next returned nil segment Aug 7 21:55:34.281113294 (12h15m33.870458281s) [upload=12734]: Waiting for error group managing segments... Aug 7 21:55:34.581166654 (12h15m34.170511631s) [noise=true , address=64.49.13.154:4935 , piece_id=O6TZCBWBHVESLRX76EFTUT4U4OIKFFVXO5UKLMWAJA2BDTHIHN2A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.075145374 (12h15m34.664490361s) [noise=true , address=79.127.201.215:4353 , piece_id=II3JPNVAATS6APC4JGPPOJRHQRMHS5EYGLLEEAMTOPHPEGRQUF7A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.117523204 (12h15m34.706868191s) [noise=true , address=77.90.31.14:4678 , piece_id=7WZVWEAFKI4ZEIHTZOJNMAQWLBVPPLB7U4JNYZQNXCVRC2S4BUBA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.189259324 (12h15m34.778604311s) [noise=true , address=104.36.82.9:4014 , piece_id=D3NLTEDBRB4M7GIMPYCDNQ66HYLAZGSSBRM7SJWU45LPTCRMYMCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.208589096 (12h15m34.797934083s) [noise=true , address=104.36.82.23:4474 , piece_id=WM2SVCDZEFAN6VXQ2PWKBZGUCHIHNJ52RNKJKUNBMJ3AHFXUT3PA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.240233681 (12h15m34.829578668s) [noise=true , address=104.36.82.58:4265 , piece_id=XMDZZK372B4DNLPJRKD2BM7OI7JNQSZ6ABLSXNPZEH2WT73AT7TA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.249509658 (12h15m34.838854645s) [noise=true , address=104.36.82.39:4739 , piece_id=QGKQ3NJO7OTN5O2QQU43KYJXARJW6JQMSCONEJR35AEBBNRORJCA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.254602933 (12h15m34.84394792s) [noise=true , address=104.36.82.12:4179 , piece_id=KFVQX32X6I7QBU552L4NSPZG3JTVT24YESONPWGM6UCTR4ZWBUKQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.255644015 (12h15m34.844989002s) [noise=true , address=104.36.82.59:4322 , piece_id=O6L7GOHWQOA6VAMNJATA232B7BEQS66XKSZXUAQLX3KYNQXTMUCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.260038844 (12h15m34.849383831s) [noise=true , address=45.91.116.8:4065 , piece_id=OEVAVKCHFYMJMGBKUKF5F34O3NH47DG45QJHHNWMOD7TDRMCQLAQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.160031235 (12h15m35.749376222s) [noise=true , address=104.36.82.43:4850 , piece_id=JH2ASD4NQAQMOBOZMDGFKVPPOS2P26QIFSYDNHYZN24RP52E4XLA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.204255979 (12h15m35.793600956s) [noise=true , address=104.36.82.62:4429 , piece_id=RCPKZHWWS56QSEVG6L25B3FMQOU5ZES3BNAEYIUYQ3PLQV3FDMVQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.608134744 (12h15m36.197479731s) [noise=true , address=104.36.82.37:4674 , piece_id=AN3RCVXXXNEJUD33D4W47FDFYJQ7X5ZK7H4BJD6NR4PJOJSCVV6A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.985735083 (12h15m36.57508007s) [noise=true , address=104.36.82.45:4926 , piece_id=S3EH75AGZ3QVGR6RM3RI3SCHXZJPE7LORX2QQA5SCEADV3RDYM6A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.985758668 (12h15m36.575103655s) [seg_pos={0 0}, upload=12734]: Segment reached optimal threshold of 14 pieces. Aug 7 21:55:36.985891332 (12h15m36.575236319s) [noise=true , address=104.36.82.36:4633 , piece_id=6RL3INCLYJCEH53JOFJDL2MMEXGQ2G4T33PKZPJ3W2FG5QMCFLCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dvBPvb9YEtM4NieVCXL71Pak5zwiMg14MSXH4RuLZts4dByG8): protocol: expected piece hash; context canceled; EOF Aug 7 21:55:36.986091084 (12h15m36.575436071s) [noise=true , address=104.36.82.47:4520 , piece_id=K75KQHBYH7QTZDCFM7ZW55EUO2W3UN2NEIKUMNCMQWHYYNOWLL3A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy): protocol: write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection; write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection Aug 7 21:55:36.986139326 (12h15m36.575484333s) [noise=true , address=104.36.82.54:4166 , piece_id=5SBUSGFVYPNFGILW6OTNLGJQG7CNI7GYYEJQJMITPKFZ2MAK5QRQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv): protocol: expected piece hash; context canceled; EOF Aug 7 21:55:36.986268794 (12h15m36.575613791s) [noise=true , address=104.36.82.44:4894 , piece_id=WCLKE2VTORDUBR5II5IQ5OTYMNHCMP2OYAIBHG3GV5P5U3FCO7HQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44): protocol: read tcp 188.166.225.170:34542->104.36.82.44:4894: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:55:36.986293281 (12h15m36.575638268s) [seg_pos={0 0}, upload=12734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:55:36.986385397 (12h15m36.575730394s) [upload=12734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:55:36.986391038 (12h15m36.575736025s) [upload=12734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:55:37.437633211 (12h15m37.026978238s) [upload=12734]: Done uploading segments... Aug 7 21:55:37.437656937 (12h15m37.027001924s) [upload=12734]: Upload finished. err: <nil> Aug 7 21:55:48.800457325 (12h15m48.389802332s) [upload=12741]: Starting upload Aug 7 21:55:48.800589529 (12h15m48.389934526s) [upload=12741]: Done starting upload Aug 7 21:55:48.800688879 (12h15m48.390033866s) [upload=12741]: Uploading segments... Aug 7 21:55:48.801167825 (12h15m48.390512802s) [upload=12741]: Got next segment. Inline: false Aug 7 21:55:48.801200477 (12h15m48.390545464s) [upload=12741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:55:48.801204775 (12h15m48.390549762s) [upload=12741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:55:49.015107648 (12h15m48.604452635s) [upload=12741]: Next segment err: metaclient: Too Many Requests Aug 7 21:55:49.015124520 (12h15m48.604469507s) [upload=12741]: Done uploading segments... Aug 7 21:55:49.015136393 (12h15m48.60448138s) [upload=12741]: Upload finished. err: metaclient: Too Many Requests Duration: 7.078706051s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 32 out of 60 is failed (threshold=3) [.......X.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..X.................] duration: 1.90395738s FAIL (2025-08-07 22:00:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6664 (11h58m33.293011651s) [upload=12450]: Starting upload Aug 7 21:38:33.703724295 (11h58m33.293069282s) [upload=12450]: Done starting upload Aug 7 21:38:33.703770663 (11h58m33.29311565s) [upload=12450]: Uploading segments... Aug 7 21:38:33.703817212 (11h58m33.293162199s) [upload=12450]: Got next segment. Inline: true Aug 7 21:38:33.703820899 (11h58m33.293165886s) [upload=12450]: Waiting for error group managing segments... Aug 7 21:38:33.703853411 (11h58m33.293198388s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.703857378 (11h58m33.293202365s) [upload=12450]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.703859744 (11h58m33.293204731s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:33.059235349 (11h59m32.648580326s) [upload=12450]: Done uploading segments... Aug 7 21:39:33.059253302 (11h59m32.648598289s) [upload=12450]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:33.956833499 (11h59m33.546178486s) [upload=12464]: Starting upload Aug 7 21:39:33.956907240 (11h59m33.546252227s) [upload=12464]: Done starting upload Aug 7 21:39:33.956967195 (11h59m33.546312182s) [upload=12464]: Uploading segments... Aug 7 21:39:33.957011540 (11h59m33.546356527s) [upload=12464]: Got next segment. Inline: true Aug 7 21:39:33.957016229 (11h59m33.546361226s) [upload=12464]: Waiting for error group managing segments... Aug 7 21:39:33.957053820 (11h59m33.546398807s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:33.957058250 (11h59m33.546403237s) [upload=12464]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:33.957061736 (11h59m33.546406723s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:33.302006486 (12h0m32.891351473s) [upload=12464]: Done uploading segments... Aug 7 21:40:33.302025733 (12h0m32.89137071s) [upload=12464]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:34.211725785 (12h0m33.801070772s) [upload=12482]: Starting upload Aug 7 21:40:34.211806970 (12h0m33.801151957s) [upload=12482]: Done starting upload Aug 7 21:40:34.211965303 (12h0m33.8013103s) [upload=12482]: Uploading segments... Aug 7 21:40:34.212066617 (12h0m33.801411614s) [upload=12482]: Got next segment. Inline: true Aug 7 21:40:34.212072819 (12h0m33.801417806s) [upload=12482]: Waiting for error group managing segments... Aug 7 21:40:34.212126081 (12h0m33.801471068s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:34.212132573 (12h0m33.80147756s) [upload=12482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:34.212136120 (12h0m33.801481107s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:33.557664583 (12h1m33.14700957s) [upload=12482]: Done uploading segments... Aug 7 21:41:33.557694410 (12h1m33.147039387s) [upload=12482]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.468736652 (12h1m34.058081639s) [upload=12501]: Starting upload Aug 7 21:41:34.468803340 (12h1m34.058148327s) [upload=12501]: Done starting upload Aug 7 21:41:34.468862643 (12h1m34.05820763s) [upload=12501]: Uploading segments... Aug 7 21:41:34.468901758 (12h1m34.058246735s) [upload=12501]: Got next segment. Inline: true Aug 7 21:41:34.468905907 (12h1m34.058250884s) [upload=12501]: Waiting for error group managing segments... Aug 7 21:41:34.468943147 (12h1m34.058288134s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.468947105 (12h1m34.058292092s) [upload=12501]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.468949690 (12h1m34.058294677s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:33.814395640 (12h2m33.403740627s) [upload=12501]: Done uploading segments... Aug 7 21:42:33.814423744 (12h2m33.403768731s) [upload=12501]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:46.083099596 (12h2m45.672444573s) [upload=12522]: Starting upload Aug 7 21:42:46.083172004 (12h2m45.672516981s) [upload=12522]: Done starting upload Aug 7 21:42:46.083231107 (12h2m45.672576094s) [upload=12522]: Uploading segments... Aug 7 21:42:46.083339093 (12h2m45.67268408s) [upload=12522]: Got next segment. Inline: true Aug 7 21:42:46.083344093 (12h2m45.67268908s) [upload=12522]: Waiting for error group managing segments... Aug 7 21:42:46.083408827 (12h2m45.672753814s) [upload=12522]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:46.083414187 (12h2m45.672759174s) [upload=12522]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:46.083416732 (12h2m45.672761719s) [upload=12522]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:47.411975663 (12h2m47.00132066s) [upload=12522]: Done uploading segments... Aug 7 21:42:47.411996032 (12h2m47.001341019s) [upload=12522]: Upload finished. err: <nil> Aug 7 21:43:11.548517821 (12h3m11.137862808s) [upload=12529]: Starting upload Aug 7 21:43:11.548568578 (12h3m11.137913565s) [upload=12529]: Done starting upload Aug 7 21:43:11.548617301 (12h3m11.137962288s) [upload=12529]: Uploading segments... Aug 7 21:43:11.548646878 (12h3m11.137991865s) [upload=12529]: Got next segment. Inline: true Aug 7 21:43:11.548650374 (12h3m11.137995362s) [upload=12529]: Waiting for error group managing segments... Aug 7 21:43:11.548681634 (12h3m11.138026621s) [upload=12529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:11.548685361 (12h3m11.138030338s) [upload=12529]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:11.548687726 (12h3m11.138032713s) [upload=12529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:10.906282559 (12h4m10.495627546s) [upload=12529]: Done uploading segments... Aug 7 21:44:10.906303038 (12h4m10.495648015s) [upload=12529]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:44:11.778510580 (12h4m11.367855567s) [upload=12548]: Starting upload Aug 7 21:44:11.778570013 (12h4m11.367915s) [upload=12548]: Done starting upload Aug 7 21:44:11.778662139 (12h4m11.368007126s) [upload=12548]: Uploading segments... Aug 7 21:44:11.778800213 (12h4m11.36814519s) [upload=12548]: Got next segment. Inline: true Aug 7 21:44:11.778804892 (12h4m11.368149869s) [upload=12548]: Waiting for error group managing segments... Aug 7 21:44:11.778861480 (12h4m11.368206467s) [upload=12548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:11.778866230 (12h4m11.368211207s) [upload=12548]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:11.778868745 (12h4m11.368213732s) [upload=12548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:11.131015060 (12h5m10.720360047s) [upload=12548]: Done uploading segments... Aug 7 21:45:11.131033725 (12h5m10.720378712s) [upload=12548]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:45:11.998653609 (12h5m11.587998596s) [upload=12563]: Starting upload Aug 7 21:45:11.998732921 (12h5m11.588077908s) [upload=12563]: Done starting upload Aug 7 21:45:11.998811802 (12h5m11.588156789s) [upload=12563]: Uploading segments... Aug 7 21:45:11.998922443 (12h5m11.58826743s) [upload=12563]: Got next segment. Inline: true Aug 7 21:45:11.998926992 (12h5m11.588271979s) [upload=12563]: Waiting for error group managing segments... Aug 7 21:45:11.998979913 (12h5m11.5883249s) [upload=12563]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:11.998987267 (12h5m11.588332254s) [upload=12563]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:11.998990603 (12h5m11.58833559s) [upload=12563]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:14.137150454 (12h5m13.726495441s) [upload=12563]: Done uploading segments... Aug 7 21:45:14.137168529 (12h5m13.726513516s) [upload=12563]: Upload finished. err: <nil> Aug 7 21:45:53.851988781 (12h5m53.441333768s) [upload=12569]: Starting upload Aug 7 21:45:53.852043686 (12h5m53.441388673s) [upload=12569]: Done starting upload Aug 7 21:45:53.852088772 (12h5m53.441433759s) [upload=12569]: Uploading segments... Aug 7 21:45:53.852141613 (12h5m53.4414866s) [upload=12569]: Got next segment. Inline: true Aug 7 21:45:53.852146192 (12h5m53.441491169s) [upload=12569]: Waiting for error group managing segments... Aug 7 21:45:53.852182612 (12h5m53.441527599s) [upload=12569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:53.852186920 (12h5m53.441531907s) [upload=12569]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:53.852189195 (12h5m53.441534182s) [upload=12569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:54.127817304 (12h5m53.717162291s) [upload=12569]: Done uploading segments... Aug 7 21:45:54.127834497 (12h5m53.717179484s) [upload=12569]: Upload finished. err: <nil> Aug 7 21:46:54.936546775 (12h6m54.525891762s) [upload=12585]: Starting upload Aug 7 21:46:54.936595597 (12h6m54.525940585s) [upload=12585]: Done starting upload Aug 7 21:46:54.936645253 (12h6m54.52599024s) [upload=12585]: Uploading segments... Aug 7 21:46:54.936675961 (12h6m54.526020948s) [upload=12585]: Got next segment. Inline: true Aug 7 21:46:54.936680381 (12h6m54.526025358s) [upload=12585]: Waiting for error group managing segments... Aug 7 21:46:54.936716089 (12h6m54.526061076s) [upload=12585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:54.936720797 (12h6m54.526065774s) [upload=12585]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:46:54.936723172 (12h6m54.526068159s) [upload=12585]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:54.288569811 (12h7m53.877914798s) [upload=12585]: Done uploading segments... Aug 7 21:47:54.288620218 (12h7m53.877965205s) [upload=12585]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.903983581s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....X...............X..........X.................X.........] duration: 2.403525391s FAIL (2025-08-07 22:00:29 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 2.403550601s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 33 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X...X......................] duration: 1.906653397s FAIL (2025-08-07 22:00:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== line exceeded Aug 7 21:37:22.572699237 (11h57m22.162044224s) [upload=12432]: Starting upload Aug 7 21:37:22.572782937 (11h57m22.162127924s) [upload=12432]: Done starting upload Aug 7 21:37:22.572842260 (11h57m22.162187247s) [upload=12432]: Uploading segments... Aug 7 21:37:22.572885313 (11h57m22.1622303s) [upload=12432]: Got next segment. Inline: true Aug 7 21:37:22.572889591 (11h57m22.162234578s) [upload=12432]: Waiting for error group managing segments... Aug 7 21:37:22.572930138 (11h57m22.162275125s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:22.572934296 (11h57m22.162279283s) [upload=12432]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:22.572937121 (11h57m22.162282118s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.933954276 (11h57m51.523299253s) [upload=12432]: Done uploading segments... Aug 7 21:37:51.933970927 (11h57m51.523315914s) [upload=12432]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:28.745940745 (11h58m28.335285732s) [upload=12448]: Starting upload Aug 7 21:38:28.745997504 (11h58m28.335342491s) [upload=12448]: Done starting upload Aug 7 21:38:28.746040176 (11h58m28.335385163s) [upload=12448]: Uploading segments... Aug 7 21:38:28.746078458 (11h58m28.335423446s) [upload=12448]: Got next segment. Inline: true Aug 7 21:38:28.746081766 (11h58m28.335426743s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:38:28.746111923 (11h58m28.33545691s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:28.746115540 (11h58m28.335460527s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:28.746118205 (11h58m28.335463192s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:58.105315693 (11h58m57.694660691s) [upload=12448]: Done uploading segments... Aug 7 21:38:58.105342465 (11h58m57.694687452s) [upload=12448]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:27.137238449 (11h59m26.726583436s) [upload=12462]: Starting upload Aug 7 21:39:27.137294907 (11h59m26.726639894s) [upload=12462]: Done starting upload Aug 7 21:39:27.137337368 (11h59m26.726682355s) [upload=12462]: Uploading segments... Aug 7 21:39:27.137366043 (11h59m26.72671103s) [upload=12462]: Got next segment. Inline: true Aug 7 21:39:27.137370061 (11h59m26.726715048s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:39:27.137401901 (11h59m26.726746879s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:27.137405518 (11h59m26.726750495s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:27.137407813 (11h59m26.72675279s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.495186447 (11h59m56.084531434s) [upload=12462]: Done uploading segments... Aug 7 21:39:56.495206415 (11h59m56.084551402s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:01.914615953 (12h0m1.50396094s) [upload=12475]: Starting upload Aug 7 21:40:01.914684295 (12h0m1.504029282s) [upload=12475]: Done starting upload Aug 7 21:40:01.914741163 (12h0m1.50408616s) [upload=12475]: Uploading segments... Aug 7 21:40:01.914783103 (12h0m1.50412809s) [upload=12475]: Got next segment. Inline: true Aug 7 21:40:01.914788143 (12h0m1.504133131s) [upload=12475]: Waiting for error group managing segments... Aug 7 21:40:01.914833950 (12h0m1.504178937s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:01.914840073 (12h0m1.50418507s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:01.914846014 (12h0m1.504191011s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:31.268196986 (12h0m30.857542023s) [upload=12475]: Done uploading segments... Aug 7 21:40:31.268223536 (12h0m30.857568533s) [upload=12475]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:46.248438399 (12h0m45.837783386s) [upload=12487]: Starting upload Aug 7 21:40:46.248491410 (12h0m45.837836387s) [upload=12487]: Done starting upload Aug 7 21:40:46.248540414 (12h0m45.837885401s) [upload=12487]: Uploading segments... Aug 7 21:40:46.248572345 (12h0m45.837917332s) [upload=12487]: Got next segment. Inline: true Aug 7 21:40:46.248576133 (12h0m45.83792112s) [upload=12487]: Waiting for error group managing segments... Aug 7 21:40:46.248613013 (12h0m45.837958s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:46.248616850 (12h0m45.837961837s) [upload=12487]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:46.248619255 (12h0m45.837964242s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:55.261300379 (12h0m54.850645366s) [upload=12487]: Done uploading segments... Aug 7 21:40:55.261318514 (12h0m54.850663501s) [upload=12487]: Upload finished. err: <nil> Aug 7 21:42:00.373248983 (12h1m59.962594s) [upload=12509]: Starting upload Aug 7 21:42:00.373363813 (12h1m59.96270884s) [upload=12509]: Done starting upload Aug 7 21:42:00.373438045 (12h1m59.962783062s) [upload=12509]: Uploading segments... Aug 7 21:42:00.373501075 (12h1m59.962846083s) [upload=12509]: Got next segment. Inline: true Aug 7 21:42:00.373507809 (12h1m59.962852816s) [upload=12509]: Waiting for error group managing segments... Aug 7 21:42:00.373577642 (12h1m59.962922669s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:00.373584325 (12h1m59.962929342s) [upload=12509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:00.373590016 (12h1m59.962935053s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:29.724044552 (12h2m29.313389539s) [upload=12509]: Done uploading segments... Aug 7 21:42:29.724072415 (12h2m29.313417392s) [upload=12509]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:30.611135457 (12h2m30.200480444s) [upload=12516]: Starting upload Aug 7 21:42:30.611201894 (12h2m30.200546881s) [upload=12516]: Done starting upload Aug 7 21:42:30.611252732 (12h2m30.200597719s) [upload=12516]: Uploading segments... Aug 7 21:42:30.611338045 (12h2m30.200683032s) [upload=12516]: Got next segment. Inline: true Aug 7 21:42:30.611344076 (12h2m30.200689063s) [upload=12516]: Waiting for error group managing segments... Aug 7 21:42:30.611400083 (12h2m30.20074507s) [upload=12516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:30.611404922 (12h2m30.200749909s) [upload=12516]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:30.611408178 (12h2m30.200753166s) [upload=12516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:46.271341803 (12h2m45.860686781s) [upload=12516]: Done uploading segments... Aug 7 21:42:46.271358525 (12h2m45.860703512s) [upload=12516]: Upload finished. err: <nil> Aug 7 21:43:05.160958738 (12h3m4.750303725s) [upload=12528]: Starting upload Aug 7 21:43:05.161064639 (12h3m4.750409626s) [upload=12528]: Done starting upload Aug 7 21:43:05.161177936 (12h3m4.750522953s) [upload=12528]: Uploading segments... Aug 7 21:43:05.161261357 (12h3m4.750606344s) [upload=12528]: Got next segment. Inline: true Aug 7 21:43:05.161266376 (12h3m4.750611353s) [upload=12528]: Waiting for error group managing segments... Aug 7 21:43:05.161306081 (12h3m4.750651069s) [upload=12528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:05.161310580 (12h3m4.750655567s) [upload=12528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:05.161312864 (12h3m4.750657841s) [upload=12528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:17.899724589 (12h3m17.489069576s) [upload=12528]: Done uploading segments... Aug 7 21:43:17.899747483 (12h3m17.48909251s) [upload=12528]: Upload finished. err: <nil> Aug 7 21:43:34.894582431 (12h3m34.483927418s) [upload=12537]: Starting upload Aug 7 21:43:34.894647576 (12h3m34.483992564s) [upload=12537]: Done starting upload Aug 7 21:43:34.894699496 (12h3m34.484044483s) [upload=12537]: Uploading segments... Aug 7 21:43:34.894732940 (12h3m34.484077927s) [upload=12537]: Got next segment. Inline: true Aug 7 21:43:34.894736967 (12h3m34.484081954s) [upload=12537]: Waiting for error group managing segments... Aug 7 21:43:34.894770662 (12h3m34.484115649s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:34.894775050 (12h3m34.484120027s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:34.894777795 (12h3m34.484122782s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:35.191263901 (12h3m34.780608888s) [upload=12537]: Done uploading segments... Aug 7 21:43:35.191280001 (12h3m34.780624978s) [upload=12537]: Upload finished. err: <nil> Aug 7 21:44:27.756622626 (12h4m27.345967613s) [upload=12553]: Starting upload Aug 7 21:44:27.756679294 (12h4m27.346024281s) [upload=12553]: Done starting upload Aug 7 21:44:27.756721205 (12h4m27.346066192s) [upload=12553]: Uploading segments... Aug 7 21:44:27.756751423 (12h4m27.34609641s) [upload=12553]: Got next segment. Inline: true Aug 7 21:44:27.756754779 (12h4m27.346099766s) [upload=12553]: Waiting for error group managing segments... Aug 7 21:44:27.756785678 (12h4m27.346130655s) [upload=12553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:27.756789615 (12h4m27.346134602s) [upload=12553]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:27.756791790 (12h4m27.346136777s) [upload=12553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:57.108386877 (12h4m56.697731864s) [upload=12553]: Done uploading segments... Aug 7 21:44:57.108405372 (12h4m56.697750359s) [upload=12553]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.906687214s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X......X..X.............X..X....................X.......] duration: 8.367867083s FAIL (2025-08-07 22:00:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 57 (12h15m32.758009944s) [noise=true , address=104.36.82.54:4166 , piece_id=5SBUSGFVYPNFGILW6OTNLGJQG7CNI7GYYEJQJMITPKFZ2MAK5QRQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.168945643 (12h15m32.75829063s) [noise=true , address=104.36.82.12:4179 , piece_id=KFVQX32X6I7QBU552L4NSPZG3JTVT24YESONPWGM6UCTR4ZWBUKQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.168966093 (12h15m32.75831109s) [noise=true , address=104.36.82.43:4850 , piece_id=JH2ASD4NQAQMOBOZMDGFKVPPOS2P26QIFSYDNHYZN24RP52E4XLA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169201393 (12h15m32.75854638s) [noise=true , address=104.36.82.36:4633 , piece_id=6RL3INCLYJCEH53JOFJDL2MMEXGQ2G4T33PKZPJ3W2FG5QMCFLCQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169453466 (12h15m32.758798453s) [noise=true , address=104.36.82.47:4520 , piece_id=K75KQHBYH7QTZDCFM7ZW55EUO2W3UN2NEIKUMNCMQWHYYNOWLL3A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169588975 (12h15m32.758933962s) [noise=true , address=104.36.82.9:4014 , piece_id=D3NLTEDBRB4M7GIMPYCDNQ66HYLAZGSSBRM7SJWU45LPTCRMYMCQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170078472 (12h15m32.759423459s) [noise=true , address=104.36.82.39:4739 , piece_id=QGKQ3NJO7OTN5O2QQU43KYJXARJW6JQMSCONEJR35AEBBNRORJCA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170313160 (12h15m32.759658147s) [noise=true , address=104.36.82.62:4429 , piece_id=RCPKZHWWS56QSEVG6L25B3FMQOU5ZES3BNAEYIUYQ3PLQV3FDMVQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170515587 (12h15m32.759860574s) [noise=true , address=79.127.201.215:4353 , piece_id=II3JPNVAATS6APC4JGPPOJRHQRMHS5EYGLLEEAMTOPHPEGRQUF7A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170753382 (12h15m32.760098359s) [noise=true , address=45.91.116.8:4065 , piece_id=OEVAVKCHFYMJMGBKUKF5F34O3NH47DG45QJHHNWMOD7TDRMCQLAQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170965789 (12h15m32.760310776s) [noise=true , address=77.90.31.14:4678 , piece_id=7WZVWEAFKI4ZEIHTZOJNMAQWLBVPPLB7U4JNYZQNXCVRC2S4BUBA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171167525 (12h15m32.760512512s) [noise=true , address=104.36.82.45:4926 , piece_id=S3EH75AGZ3QVGR6RM3RI3SCHXZJPE7LORX2QQA5SCEADV3RDYM6A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171354372 (12h15m32.760699359s) [noise=true , address=64.49.13.154:4935 , piece_id=O6TZCBWBHVESLRX76EFTUT4U4OIKFFVXO5UKLMWAJA2BDTHIHN2A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171604611 (12h15m32.760949598s) [noise=true , address=104.36.82.44:4894 , piece_id=WCLKE2VTORDUBR5II5IQ5OTYMNHCMP2OYAIBHG3GV5P5U3FCO7HQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171788884 (12h15m32.761133861s) [noise=true , address=104.36.82.37:4674 , piece_id=AN3RCVXXXNEJUD33D4W47FDFYJQ7X5ZK7H4BJD6NR4PJOJSCVV6A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.181382449 (12h15m32.770727426s) [noise=true , address=104.36.82.23:4474 , piece_id=WM2SVCDZEFAN6VXQ2PWKBZGUCHIHNJ52RNKJKUNBMJ3AHFXUT3PA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:34.281100029 (12h15m33.870445006s) [upload=12734]: Next returned nil segment Aug 7 21:55:34.281113294 (12h15m33.870458281s) [upload=12734]: Waiting for error group managing segments... Aug 7 21:55:34.581166654 (12h15m34.170511631s) [noise=true , address=64.49.13.154:4935 , piece_id=O6TZCBWBHVESLRX76EFTUT4U4OIKFFVXO5UKLMWAJA2BDTHIHN2A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.075145374 (12h15m34.664490361s) [noise=true , address=79.127.201.215:4353 , piece_id=II3JPNVAATS6APC4JGPPOJRHQRMHS5EYGLLEEAMTOPHPEGRQUF7A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.117523204 (12h15m34.706868191s) [noise=true , address=77.90.31.14:4678 , piece_id=7WZVWEAFKI4ZEIHTZOJNMAQWLBVPPLB7U4JNYZQNXCVRC2S4BUBA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.189259324 (12h15m34.778604311s) [noise=true , address=104.36.82.9:4014 , piece_id=D3NLTEDBRB4M7GIMPYCDNQ66HYLAZGSSBRM7SJWU45LPTCRMYMCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.208589096 (12h15m34.797934083s) [noise=true , address=104.36.82.23:4474 , piece_id=WM2SVCDZEFAN6VXQ2PWKBZGUCHIHNJ52RNKJKUNBMJ3AHFXUT3PA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.240233681 (12h15m34.829578668s) [noise=true , address=104.36.82.58:4265 , piece_id=XMDZZK372B4DNLPJRKD2BM7OI7JNQSZ6ABLSXNPZEH2WT73AT7TA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.249509658 (12h15m34.838854645s) [noise=true , address=104.36.82.39:4739 , piece_id=QGKQ3NJO7OTN5O2QQU43KYJXARJW6JQMSCONEJR35AEBBNRORJCA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.254602933 (12h15m34.84394792s) [noise=true , address=104.36.82.12:4179 , piece_id=KFVQX32X6I7QBU552L4NSPZG3JTVT24YESONPWGM6UCTR4ZWBUKQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.255644015 (12h15m34.844989002s) [noise=true , address=104.36.82.59:4322 , piece_id=O6L7GOHWQOA6VAMNJATA232B7BEQS66XKSZXUAQLX3KYNQXTMUCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.260038844 (12h15m34.849383831s) [noise=true , address=45.91.116.8:4065 , piece_id=OEVAVKCHFYMJMGBKUKF5F34O3NH47DG45QJHHNWMOD7TDRMCQLAQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.160031235 (12h15m35.749376222s) [noise=true , address=104.36.82.43:4850 , piece_id=JH2ASD4NQAQMOBOZMDGFKVPPOS2P26QIFSYDNHYZN24RP52E4XLA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.204255979 (12h15m35.793600956s) [noise=true , address=104.36.82.62:4429 , piece_id=RCPKZHWWS56QSEVG6L25B3FMQOU5ZES3BNAEYIUYQ3PLQV3FDMVQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.608134744 (12h15m36.197479731s) [noise=true , address=104.36.82.37:4674 , piece_id=AN3RCVXXXNEJUD33D4W47FDFYJQ7X5ZK7H4BJD6NR4PJOJSCVV6A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.985735083 (12h15m36.57508007s) [noise=true , address=104.36.82.45:4926 , piece_id=S3EH75AGZ3QVGR6RM3RI3SCHXZJPE7LORX2QQA5SCEADV3RDYM6A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.985758668 (12h15m36.575103655s) [seg_pos={0 0}, upload=12734]: Segment reached optimal threshold of 14 pieces. Aug 7 21:55:36.985891332 (12h15m36.575236319s) [noise=true , address=104.36.82.36:4633 , piece_id=6RL3INCLYJCEH53JOFJDL2MMEXGQ2G4T33PKZPJ3W2FG5QMCFLCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dvBPvb9YEtM4NieVCXL71Pak5zwiMg14MSXH4RuLZts4dByG8): protocol: expected piece hash; context canceled; EOF Aug 7 21:55:36.986091084 (12h15m36.575436071s) [noise=true , address=104.36.82.47:4520 , piece_id=K75KQHBYH7QTZDCFM7ZW55EUO2W3UN2NEIKUMNCMQWHYYNOWLL3A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy): protocol: write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection; write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection Aug 7 21:55:36.986139326 (12h15m36.575484333s) [noise=true , address=104.36.82.54:4166 , piece_id=5SBUSGFVYPNFGILW6OTNLGJQG7CNI7GYYEJQJMITPKFZ2MAK5QRQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv): protocol: expected piece hash; context canceled; EOF Aug 7 21:55:36.986268794 (12h15m36.575613791s) [noise=true , address=104.36.82.44:4894 , piece_id=WCLKE2VTORDUBR5II5IQ5OTYMNHCMP2OYAIBHG3GV5P5U3FCO7HQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44): protocol: read tcp 188.166.225.170:34542->104.36.82.44:4894: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:55:36.986293281 (12h15m36.575638268s) [seg_pos={0 0}, upload=12734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:55:36.986385397 (12h15m36.575730394s) [upload=12734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:55:36.986391038 (12h15m36.575736025s) [upload=12734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:55:37.437633211 (12h15m37.026978238s) [upload=12734]: Done uploading segments... Aug 7 21:55:37.437656937 (12h15m37.027001924s) [upload=12734]: Upload finished. err: <nil> Aug 7 21:55:48.800457325 (12h15m48.389802332s) [upload=12741]: Starting upload Aug 7 21:55:48.800589529 (12h15m48.389934526s) [upload=12741]: Done starting upload Aug 7 21:55:48.800688879 (12h15m48.390033866s) [upload=12741]: Uploading segments... Aug 7 21:55:48.801167825 (12h15m48.390512802s) [upload=12741]: Got next segment. Inline: false Aug 7 21:55:48.801200477 (12h15m48.390545464s) [upload=12741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:55:48.801204775 (12h15m48.390549762s) [upload=12741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:55:49.015107648 (12h15m48.604452635s) [upload=12741]: Next segment err: metaclient: Too Many Requests Aug 7 21:55:49.015124520 (12h15m48.604469507s) [upload=12741]: Done uploading segments... Aug 7 21:55:49.015136393 (12h15m48.60448138s) [upload=12741]: Upload finished. err: metaclient: Too Many Requests Duration: 8.367902089s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 33 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X..........................] duration: 2.28984368s FAIL (2025-08-07 22:00:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.289868417s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [............X....X.X...X....................................] duration: 745.556896ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:00:10 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 745.59019ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X.................] duration: 6.276817046s FAIL (2025-08-07 22:00:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.276843326s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 32 out of 60 is failed (threshold=3) [........X.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..X................] duration: 1.903531252s FAIL (2025-08-07 21:59:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6664 (11h58m33.293011651s) [upload=12450]: Starting upload Aug 7 21:38:33.703724295 (11h58m33.293069282s) [upload=12450]: Done starting upload Aug 7 21:38:33.703770663 (11h58m33.29311565s) [upload=12450]: Uploading segments... Aug 7 21:38:33.703817212 (11h58m33.293162199s) [upload=12450]: Got next segment. Inline: true Aug 7 21:38:33.703820899 (11h58m33.293165886s) [upload=12450]: Waiting for error group managing segments... Aug 7 21:38:33.703853411 (11h58m33.293198388s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.703857378 (11h58m33.293202365s) [upload=12450]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.703859744 (11h58m33.293204731s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:33.059235349 (11h59m32.648580326s) [upload=12450]: Done uploading segments... Aug 7 21:39:33.059253302 (11h59m32.648598289s) [upload=12450]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:33.956833499 (11h59m33.546178486s) [upload=12464]: Starting upload Aug 7 21:39:33.956907240 (11h59m33.546252227s) [upload=12464]: Done starting upload Aug 7 21:39:33.956967195 (11h59m33.546312182s) [upload=12464]: Uploading segments... Aug 7 21:39:33.957011540 (11h59m33.546356527s) [upload=12464]: Got next segment. Inline: true Aug 7 21:39:33.957016229 (11h59m33.546361226s) [upload=12464]: Waiting for error group managing segments... Aug 7 21:39:33.957053820 (11h59m33.546398807s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:33.957058250 (11h59m33.546403237s) [upload=12464]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:33.957061736 (11h59m33.546406723s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:33.302006486 (12h0m32.891351473s) [upload=12464]: Done uploading segments... Aug 7 21:40:33.302025733 (12h0m32.89137071s) [upload=12464]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:34.211725785 (12h0m33.801070772s) [upload=12482]: Starting upload Aug 7 21:40:34.211806970 (12h0m33.801151957s) [upload=12482]: Done starting upload Aug 7 21:40:34.211965303 (12h0m33.8013103s) [upload=12482]: Uploading segments... Aug 7 21:40:34.212066617 (12h0m33.801411614s) [upload=12482]: Got next segment. Inline: true Aug 7 21:40:34.212072819 (12h0m33.801417806s) [upload=12482]: Waiting for error group managing segments... Aug 7 21:40:34.212126081 (12h0m33.801471068s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:34.212132573 (12h0m33.80147756s) [upload=12482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:34.212136120 (12h0m33.801481107s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:33.557664583 (12h1m33.14700957s) [upload=12482]: Done uploading segments... Aug 7 21:41:33.557694410 (12h1m33.147039387s) [upload=12482]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.468736652 (12h1m34.058081639s) [upload=12501]: Starting upload Aug 7 21:41:34.468803340 (12h1m34.058148327s) [upload=12501]: Done starting upload Aug 7 21:41:34.468862643 (12h1m34.05820763s) [upload=12501]: Uploading segments... Aug 7 21:41:34.468901758 (12h1m34.058246735s) [upload=12501]: Got next segment. Inline: true Aug 7 21:41:34.468905907 (12h1m34.058250884s) [upload=12501]: Waiting for error group managing segments... Aug 7 21:41:34.468943147 (12h1m34.058288134s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.468947105 (12h1m34.058292092s) [upload=12501]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.468949690 (12h1m34.058294677s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:33.814395640 (12h2m33.403740627s) [upload=12501]: Done uploading segments... Aug 7 21:42:33.814423744 (12h2m33.403768731s) [upload=12501]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:46.083099596 (12h2m45.672444573s) [upload=12522]: Starting upload Aug 7 21:42:46.083172004 (12h2m45.672516981s) [upload=12522]: Done starting upload Aug 7 21:42:46.083231107 (12h2m45.672576094s) [upload=12522]: Uploading segments... Aug 7 21:42:46.083339093 (12h2m45.67268408s) [upload=12522]: Got next segment. Inline: true Aug 7 21:42:46.083344093 (12h2m45.67268908s) [upload=12522]: Waiting for error group managing segments... Aug 7 21:42:46.083408827 (12h2m45.672753814s) [upload=12522]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:46.083414187 (12h2m45.672759174s) [upload=12522]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:46.083416732 (12h2m45.672761719s) [upload=12522]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:47.411975663 (12h2m47.00132066s) [upload=12522]: Done uploading segments... Aug 7 21:42:47.411996032 (12h2m47.001341019s) [upload=12522]: Upload finished. err: <nil> Aug 7 21:43:11.548517821 (12h3m11.137862808s) [upload=12529]: Starting upload Aug 7 21:43:11.548568578 (12h3m11.137913565s) [upload=12529]: Done starting upload Aug 7 21:43:11.548617301 (12h3m11.137962288s) [upload=12529]: Uploading segments... Aug 7 21:43:11.548646878 (12h3m11.137991865s) [upload=12529]: Got next segment. Inline: true Aug 7 21:43:11.548650374 (12h3m11.137995362s) [upload=12529]: Waiting for error group managing segments... Aug 7 21:43:11.548681634 (12h3m11.138026621s) [upload=12529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:11.548685361 (12h3m11.138030338s) [upload=12529]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:11.548687726 (12h3m11.138032713s) [upload=12529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:10.906282559 (12h4m10.495627546s) [upload=12529]: Done uploading segments... Aug 7 21:44:10.906303038 (12h4m10.495648015s) [upload=12529]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:44:11.778510580 (12h4m11.367855567s) [upload=12548]: Starting upload Aug 7 21:44:11.778570013 (12h4m11.367915s) [upload=12548]: Done starting upload Aug 7 21:44:11.778662139 (12h4m11.368007126s) [upload=12548]: Uploading segments... Aug 7 21:44:11.778800213 (12h4m11.36814519s) [upload=12548]: Got next segment. Inline: true Aug 7 21:44:11.778804892 (12h4m11.368149869s) [upload=12548]: Waiting for error group managing segments... Aug 7 21:44:11.778861480 (12h4m11.368206467s) [upload=12548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:11.778866230 (12h4m11.368211207s) [upload=12548]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:11.778868745 (12h4m11.368213732s) [upload=12548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:11.131015060 (12h5m10.720360047s) [upload=12548]: Done uploading segments... Aug 7 21:45:11.131033725 (12h5m10.720378712s) [upload=12548]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:45:11.998653609 (12h5m11.587998596s) [upload=12563]: Starting upload Aug 7 21:45:11.998732921 (12h5m11.588077908s) [upload=12563]: Done starting upload Aug 7 21:45:11.998811802 (12h5m11.588156789s) [upload=12563]: Uploading segments... Aug 7 21:45:11.998922443 (12h5m11.58826743s) [upload=12563]: Got next segment. Inline: true Aug 7 21:45:11.998926992 (12h5m11.588271979s) [upload=12563]: Waiting for error group managing segments... Aug 7 21:45:11.998979913 (12h5m11.5883249s) [upload=12563]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:11.998987267 (12h5m11.588332254s) [upload=12563]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:11.998990603 (12h5m11.58833559s) [upload=12563]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:14.137150454 (12h5m13.726495441s) [upload=12563]: Done uploading segments... Aug 7 21:45:14.137168529 (12h5m13.726513516s) [upload=12563]: Upload finished. err: <nil> Aug 7 21:45:53.851988781 (12h5m53.441333768s) [upload=12569]: Starting upload Aug 7 21:45:53.852043686 (12h5m53.441388673s) [upload=12569]: Done starting upload Aug 7 21:45:53.852088772 (12h5m53.441433759s) [upload=12569]: Uploading segments... Aug 7 21:45:53.852141613 (12h5m53.4414866s) [upload=12569]: Got next segment. Inline: true Aug 7 21:45:53.852146192 (12h5m53.441491169s) [upload=12569]: Waiting for error group managing segments... Aug 7 21:45:53.852182612 (12h5m53.441527599s) [upload=12569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:53.852186920 (12h5m53.441531907s) [upload=12569]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:53.852189195 (12h5m53.441534182s) [upload=12569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:54.127817304 (12h5m53.717162291s) [upload=12569]: Done uploading segments... Aug 7 21:45:54.127834497 (12h5m53.717179484s) [upload=12569]: Upload finished. err: <nil> Aug 7 21:46:54.936546775 (12h6m54.525891762s) [upload=12585]: Starting upload Aug 7 21:46:54.936595597 (12h6m54.525940585s) [upload=12585]: Done starting upload Aug 7 21:46:54.936645253 (12h6m54.52599024s) [upload=12585]: Uploading segments... Aug 7 21:46:54.936675961 (12h6m54.526020948s) [upload=12585]: Got next segment. Inline: true Aug 7 21:46:54.936680381 (12h6m54.526025358s) [upload=12585]: Waiting for error group managing segments... Aug 7 21:46:54.936716089 (12h6m54.526061076s) [upload=12585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:54.936720797 (12h6m54.526065774s) [upload=12585]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:46:54.936723172 (12h6m54.526068159s) [upload=12585]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:54.288569811 (12h7m53.877914798s) [upload=12585]: Done uploading segments... Aug 7 21:47:54.288620218 (12h7m53.877965205s) [upload=12585]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.903553957s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [......................XX.XX..X.XX...........................] duration: 236.403812ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:59:56 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 236.429502ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXX...X..X.................] duration: 5.655689596s FAIL (2025-08-07 21:59:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.65570664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 26 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXX.....X.................] duration: 6.400313467s FAIL (2025-08-07 21:59:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== t segment. Inline: false Aug 7 21:44:25.061589945 (12h4m24.650934922s) [upload=12552]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:25.061594724 (12h4m24.650939711s) [upload=12552]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:44:50.602885060 (12h4m50.192230047s) [seg_pos={0 0}, upload=12552]: Begin upload segment... Aug 7 21:44:50.602935446 (12h4m50.192280423s) [seg_pos={0 0}, upload=12552]: Done begin upload segment. Aug 7 21:44:50.603112565 (12h4m50.192457552s) [noise=true , address=79.127.205.233:4624 , piece_id=WEAYHGF7VE6W5ML3QAE3AITMEJGQFH4JQZ7RQKR7O4NKTPR6RZRA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.603695610 (12h4m50.193040597s) [noise=true , address=104.36.82.22:4529 , piece_id=ZTTBM5H6MHVMQ3OFVTVISN6WJ7WB3XBWI7YZKUXCFNQ6NO3D4CGA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.604300768 (12h4m50.193645755s) [noise=true , address=45.91.116.6:4002 , piece_id=AE5EL7NWD6FEQ7NXX2JRHB4MVZQDXRGQU3EER2X7COISFAX6P3XQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.604588519 (12h4m50.193933496s) [noise=true , address=104.36.82.56:4249 , piece_id=RD4B7IDIJV25L7EJNSYCZGW5QKJEM435E2N3UFUD44Q4S3DCBOXQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605153860 (12h4m50.194498837s) [noise=true , address=77.90.31.11:4562 , piece_id=XC6F2WLSXKQB7E7MBRNZBJKUSUFL6XNA5TVX5M3ZJI3ZZMK7UBLA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605455377 (12h4m50.194800364s) [noise=true , address=104.36.82.23:4466 , piece_id=5J5X2LSP6GCX2PNRIDZYFJPISWSX3YSRID5M2WU6Y5Z6LWRHLC4Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605743810 (12h4m50.195088797s) [noise=true , address=104.36.82.62:4420 , piece_id=UOO36PYVE5WFRRZLGYLJ6I6SV4EO7W2ZH2TIXJ2OIZZMY3LTHR2A, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.606547457 (12h4m50.195892444s) [noise=true , address=104.36.82.5:4842 , piece_id=E5DSJXN63RFX7EQ4ZTBMUESFW77EB3HQNA5KCQGRVRAA5XO2GN3Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.606982099 (12h4m50.196327086s) [noise=true , address=104.36.82.59:4301 , piece_id=VVR5CQWCPNPZUFEFCPB6ZL2KASWS5XO65FVDI474RKRZSJFDUQTA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.607295849 (12h4m50.196640836s) [noise=true , address=45.91.116.14:4198 , piece_id=2HRU3JZDQSNGD5NJZDLZSRG2WIO2Y2QBSOAY6KUEGUKDBWM53RSA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.607695593 (12h4m50.19704058s) [noise=true , address=104.36.82.6:4881 , piece_id=VUC33D7TLVFICDLA3334DEKBJE5E5P4HQTJU3VDQNMVYWVESSJWA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.608364103 (12h4m50.19770909s) [noise=true , address=104.36.82.2:4757 , piece_id=H6KNYP5HASPWQIHADTMEZDC5AE5HITCAY73XGNWVEXJOPDEZ3SHA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.608764288 (12h4m50.198109265s) [noise=true , address=104.36.82.46:4493 , piece_id=PS4S4OZ6HSITEJFPJDIBWQ5RGWTDRKMX3NCEWXCLC2PSP5Q2KPZQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609065284 (12h4m50.198410271s) [noise=true , address=104.36.82.47:4518 , piece_id=KJ3PJCT3EDH3WZBT7AWWKI6ABGVVAMKZTVJY7Q76FULC72TUWA6Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609312777 (12h4m50.198657755s) [noise=true , address=104.36.82.60:4349 , piece_id=GTTF32YMJWBY2GZEFZPIBZXZWNI7MBLUYCTQ3SRFG735PLWXTXBQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609589888 (12h4m50.198934875s) [noise=true , address=104.36.82.52:4116 , piece_id=7QA2Q5ACN43TQUOKWA34EZFSSQBE3OWNJTYODHHQ5GCVOA4QZOJA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.610295969 (12h4m50.199641056s) [noise=true , address=104.36.82.37:4678 , piece_id=VJO5JO3SJEXBQZC6PJ34DRKWEQKOPI33ZF4IHNNOOTSD7IBRR4PQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.610811305 (12h4m50.200156302s) [noise=true , address=64.49.13.169:4714 , piece_id=A6O3H7QLE6KQTHWUKOEROVHPEHVLG22KNF2KCV3HK2KCLXP7SYVA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:51.715421583 (12h4m51.30476657s) [upload=12552]: Next returned nil segment Aug 7 21:44:51.715438125 (12h4m51.304783112s) [upload=12552]: Waiting for error group managing segments... Aug 7 21:44:51.998131281 (12h4m51.587476268s) [noise=true , address=64.49.13.169:4714 , piece_id=A6O3H7QLE6KQTHWUKOEROVHPEHVLG22KNF2KCV3HK2KCLXP7SYVA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:51.999236335 (12h4m51.588581312s) [noise=true , address=79.127.205.233:4624 , piece_id=WEAYHGF7VE6W5ML3QAE3AITMEJGQFH4JQZ7RQKR7O4NKTPR6RZRA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.638299991 (12h4m52.227644978s) [noise=true , address=45.91.116.6:4002 , piece_id=AE5EL7NWD6FEQ7NXX2JRHB4MVZQDXRGQU3EER2X7COISFAX6P3XQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.639927565 (12h4m52.229272542s) [noise=true , address=104.36.82.6:4881 , piece_id=VUC33D7TLVFICDLA3334DEKBJE5E5P4HQTJU3VDQNMVYWVESSJWA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.643220585 (12h4m52.232565572s) [noise=true , address=104.36.82.2:4757 , piece_id=H6KNYP5HASPWQIHADTMEZDC5AE5HITCAY73XGNWVEXJOPDEZ3SHA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.711872926 (12h4m52.301217913s) [noise=true , address=104.36.82.56:4249 , piece_id=RD4B7IDIJV25L7EJNSYCZGW5QKJEM435E2N3UFUD44Q4S3DCBOXQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.763601896 (12h4m52.352946883s) [noise=true , address=104.36.82.22:4529 , piece_id=ZTTBM5H6MHVMQ3OFVTVISN6WJ7WB3XBWI7YZKUXCFNQ6NO3D4CGA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.768238797 (12h4m52.357583785s) [noise=true , address=104.36.82.60:4349 , piece_id=GTTF32YMJWBY2GZEFZPIBZXZWNI7MBLUYCTQ3SRFG735PLWXTXBQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.798543408 (12h4m52.387888395s) [noise=true , address=45.91.116.14:4198 , piece_id=2HRU3JZDQSNGD5NJZDLZSRG2WIO2Y2QBSOAY6KUEGUKDBWM53RSA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.861192704 (12h4m52.450537691s) [noise=true , address=77.90.31.11:4562 , piece_id=XC6F2WLSXKQB7E7MBRNZBJKUSUFL6XNA5TVX5M3ZJI3ZZMK7UBLA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:53.070218164 (12h4m52.659563151s) [noise=true , address=104.36.82.59:4301 , piece_id=VVR5CQWCPNPZUFEFCPB6ZL2KASWS5XO65FVDI474RKRZSJFDUQTA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:53.558930346 (12h4m53.148275333s) [noise=true , address=104.36.82.46:4493 , piece_id=PS4S4OZ6HSITEJFPJDIBWQ5RGWTDRKMX3NCEWXCLC2PSP5Q2KPZQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.132342777 (12h4m54.721687765s) [noise=true , address=104.36.82.37:4678 , piece_id=VJO5JO3SJEXBQZC6PJ34DRKWEQKOPI33ZF4IHNNOOTSD7IBRR4PQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.265236558 (12h4m54.854581545s) [noise=true , address=104.36.82.52:4116 , piece_id=7QA2Q5ACN43TQUOKWA34EZFSSQBE3OWNJTYODHHQ5GCVOA4QZOJA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.265254161 (12h4m54.854599138s) [seg_pos={0 0}, upload=12552]: Segment reached optimal threshold of 14 pieces. Aug 7 21:44:55.265482308 (12h4m54.854827295s) [noise=true , address=104.36.82.5:4842 , piece_id=E5DSJXN63RFX7EQ4ZTBMUESFW77EB3HQNA5KCQGRVRAA5XO2GN3Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn): protocol: expected piece hash; write tcp 188.166.225.170:57020->104.36.82.5:4842: use of closed network connection; write tcp 188.166.225.170:57020->104.36.82.5:4842: use of closed network connection; EOF Aug 7 21:44:55.265551430 (12h4m54.854896417s) [noise=true , address=104.36.82.47:4518 , piece_id=KJ3PJCT3EDH3WZBT7AWWKI6ABGVVAMKZTVJY7Q76FULC72TUWA6Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: expected piece hash; context canceled; EOF Aug 7 21:44:55.265766391 (12h4m54.855111378s) [noise=true , address=104.36.82.23:4466 , piece_id=5J5X2LSP6GCX2PNRIDZYFJPISWSX3YSRID5M2WU6Y5Z6LWRHLC4Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rkNnk9RUhKBJ3EsakqMRb3wYwrhRLhB4ZYbrdfKPDk8erRJpP): protocol: write tcp 188.166.225.170:49724->104.36.82.23:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:44:55.265824172 (12h4m54.855169159s) [noise=true , address=104.36.82.62:4420 , piece_id=UOO36PYVE5WFRRZLGYLJ6I6SV4EO7W2ZH2TIXJ2OIZZMY3LTHR2A, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SWVQwL5PxDLaSXADwTrAYS2hVcNBuYAA6ReWuRM3UZa57QyYi): protocol: read tcp 188.166.225.170:33092->104.36.82.62:4420: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:44:55.265833430 (12h4m54.855178417s) [seg_pos={0 0}, upload=12552]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:44:55.265888796 (12h4m54.855233783s) [upload=12552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:44:55.265892303 (12h4m54.85523728s) [upload=12552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:46.088275738 (12h8m45.677620725s) [upload=12552]: Done uploading segments... Aug 7 21:48:46.088301307 (12h8m45.677646284s) [upload=12552]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.40033562s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.X.X................] duration: 6.531346156s FAIL (2025-08-07 21:59:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:46242->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:46242->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46242->104.36.82.11:4115: use of closed network connection Aug 7 21:44:51.028127350 (12h4m50.617472327s) [seg_pos={0 0}, upload=12555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:44:51.028226048 (12h4m50.617571035s) [upload=12555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:44:51.028237751 (12h4m50.617582738s) [upload=12555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:51.322864709 (12h4m50.912209697s) [upload=12555]: Done uploading segments... Aug 7 21:44:51.322882523 (12h4m50.9122275s) [upload=12555]: Upload finished. err: <nil> Aug 7 21:45:58.815791269 (12h5m58.405136256s) [upload=12571]: Starting upload Aug 7 21:45:58.815846996 (12h5m58.405191983s) [upload=12571]: Done starting upload Aug 7 21:45:58.815953540 (12h5m58.405298527s) [upload=12571]: Uploading segments... Aug 7 21:45:58.816065333 (12h5m58.405410321s) [upload=12571]: Got next segment. Inline: false Aug 7 21:45:58.816100661 (12h5m58.405445638s) [upload=12571]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:58.816106212 (12h5m58.405451199s) [upload=12571]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:45:59.064845605 (12h5m58.654190592s) [seg_pos={0 0}, upload=12571]: Begin upload segment... Aug 7 21:45:59.064885521 (12h5m58.654230508s) [seg_pos={0 0}, upload=12571]: Done begin upload segment. Aug 7 21:45:59.064910399 (12h5m58.654255376s) [noise=true , address=77.90.31.18:4807 , piece_id=63E5KXGYEJHGHYE4SKTNTMED5R4HSWIST4CR4EV6ZGON3YHV73AA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.064952560 (12h5m58.654297547s) [noise=true , address=45.91.116.12:4227 , piece_id=MDQGJ65RT4QEUFX6OFXPEAOFSTG2VXBRC23DXVVD2IV6V52UFF4Q, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065245920 (12h5m58.654590907s) [noise=true , address=45.91.116.8:4048 , piece_id=2LZRN76O3QS6FODVI52DIIMK7Z77QQBGIXYUAKZCSFI7WMGRTDUA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065360070 (12h5m58.654705057s) [noise=true , address=45.91.116.9:4262 , piece_id=D4ABUUAX4MHYLFPR6LN2REDC6VRT2W7D4VZHHYM7GZ6HWOWMG2LA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065517019 (12h5m58.654861996s) [noise=true , address=77.90.31.19:4843 , piece_id=S73TOGLK5HDCBE4BINZ72A3URDR6TUD43IVIFDY5V37EVCN6222A, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065615889 (12h5m58.654960876s) [noise=true , address=104.36.82.6:4875 , piece_id=W43EX62262JONZAGLUREOG3EGMICRBIVX5O445J3OQDQ2O475J7A, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065778009 (12h5m58.655122986s) [noise=true , address=104.36.82.11:4115 , piece_id=AH7EPSWFJFX7XDYUQOFCNXEHKQ5AMQX22NY6OFBMMHATCOVH4YJA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065884293 (12h5m58.65522928s) [noise=true , address=45.91.116.8:4047 , piece_id=TEMT2SH64IMIPQ3GDE5XCF2R2XLY33HUKRFF6KUWEYH7CDVYD4YQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066024761 (12h5m58.655369748s) [noise=true , address=77.90.31.11:4554 , piece_id=LMSB62TMAJONFULH4DAOTPWQMGLUFRZYKVG4HNJ4AUBP7EBP5FFQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066243109 (12h5m58.655588096s) [noise=true , address=45.91.116.6:4973 , piece_id=7MZSPAEXC3Q7W7U3QMZUQRUI3OHPQI3XRAJW4OGU7UZ73FEI4LIQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066452850 (12h5m58.655797837s) [noise=true , address=104.36.82.6:4874 , piece_id=T3W6RCAW4YPEU42CESOSUSFSNPDBFXR74WZUYQ2AHJNBZV776MVQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066669685 (12h5m58.656014672s) [noise=true , address=45.91.116.6:4975 , piece_id=CI7SM56L5KPNQEV52PLIETEBQVH5CFXTKVQOQIZ73YOPEGYC3KOA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066955041 (12h5m58.656300028s) [noise=true , address=104.36.82.7:4909 , piece_id=EKLBHF6P5BXX5VLSLL4QEOJXQXWM7MCOY67W5OON2UHRNMKOQFAA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.067149393 (12h5m58.65649438s) [noise=true , address=45.91.116.13:4154 , piece_id=LDSWQ62RRGHTT544S6463WLDKW3WFBWIFQGQDHLYWJJD2ESMYJNQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.067334066 (12h5m58.656679053s) [noise=true , address=104.36.82.16:4294 , piece_id=KDCYWDEXSTKCHR23CAOKKTYUMRQS5Y4LH6KIJPQFF737WA5TST7Q, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.067527215 (12h5m58.656872202s) [noise=true , address=77.90.31.15:4698 , piece_id=6L3IAVPA4PQGSMKOGSKWYG4NLM6TSA2YSIAM6S6MYSDUW26CP5CA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:46:00.855806043 (12h6m0.44515103s) [upload=12571]: Next returned nil segment Aug 7 21:46:00.855821061 (12h6m0.445166048s) [upload=12571]: Waiting for error group managing segments... Aug 7 21:46:01.180369497 (12h6m0.769714484s) [noise=true , address=77.90.31.15:4698 , piece_id=6L3IAVPA4PQGSMKOGSKWYG4NLM6TSA2YSIAM6S6MYSDUW26CP5CA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.200492297 (12h6m0.789837284s) [noise=true , address=45.91.116.6:4975 , piece_id=CI7SM56L5KPNQEV52PLIETEBQVH5CFXTKVQOQIZ73YOPEGYC3KOA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.210262691 (12h6m0.799607678s) [noise=true , address=104.36.82.16:4294 , piece_id=KDCYWDEXSTKCHR23CAOKKTYUMRQS5Y4LH6KIJPQFF737WA5TST7Q, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.220625478 (12h6m0.809970465s) [noise=true , address=104.36.82.7:4909 , piece_id=EKLBHF6P5BXX5VLSLL4QEOJXQXWM7MCOY67W5OON2UHRNMKOQFAA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.304447281 (12h6m0.893792308s) [noise=true , address=104.36.82.11:4115 , piece_id=AH7EPSWFJFX7XDYUQOFCNXEHKQ5AMQX22NY6OFBMMHATCOVH4YJA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.374462398 (12h6m0.963807385s) [noise=true , address=45.91.116.8:4047 , piece_id=TEMT2SH64IMIPQ3GDE5XCF2R2XLY33HUKRFF6KUWEYH7CDVYD4YQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.378351098 (12h6m0.967696085s) [noise=true , address=104.36.82.6:4875 , piece_id=W43EX62262JONZAGLUREOG3EGMICRBIVX5O445J3OQDQ2O475J7A, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.395257154 (12h6m0.984602141s) [noise=true , address=45.91.116.6:4973 , piece_id=7MZSPAEXC3Q7W7U3QMZUQRUI3OHPQI3XRAJW4OGU7UZ73FEI4LIQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.404312138 (12h6m0.993657125s) [noise=true , address=77.90.31.19:4843 , piece_id=S73TOGLK5HDCBE4BINZ72A3URDR6TUD43IVIFDY5V37EVCN6222A, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.409522908 (12h6m0.998867885s) [noise=true , address=45.91.116.13:4154 , piece_id=LDSWQ62RRGHTT544S6463WLDKW3WFBWIFQGQDHLYWJJD2ESMYJNQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.416315134 (12h6m1.005660121s) [noise=true , address=45.91.116.12:4227 , piece_id=MDQGJ65RT4QEUFX6OFXPEAOFSTG2VXBRC23DXVVD2IV6V52UFF4Q, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.421432725 (12h6m1.010777712s) [noise=true , address=45.91.116.8:4048 , piece_id=2LZRN76O3QS6FODVI52DIIMK7Z77QQBGIXYUAKZCSFI7WMGRTDUA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.439709112 (12h6m1.029054099s) [noise=true , address=45.91.116.9:4262 , piece_id=D4ABUUAX4MHYLFPR6LN2REDC6VRT2W7D4VZHHYM7GZ6HWOWMG2LA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.439735472 (12h6m1.029080459s) [seg_pos={0 0}, upload=12571]: Segment reached optimal threshold of 13 pieces. Aug 7 21:46:01.440013234 (12h6m1.029358221s) [noise=true , address=77.90.31.11:4554 , piece_id=LMSB62TMAJONFULH4DAOTPWQMGLUFRZYKVG4HNJ4AUBP7EBP5FFQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:01.440020518 (12h6m1.029365505s) [noise=true , address=104.36.82.6:4874 , piece_id=T3W6RCAW4YPEU42CESOSUSFSNPDBFXR74WZUYQ2AHJNBZV776MVQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: write tcp 188.166.225.170:54068->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:01.440109398 (12h6m1.029454385s) [noise=true , address=77.90.31.18:4807 , piece_id=63E5KXGYEJHGHYE4SKTNTMED5R4HSWIST4CR4EV6ZGON3YHV73AA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:01.440122312 (12h6m1.029467299s) [seg_pos={0 0}, upload=12571]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:46:01.440193058 (12h6m1.029538045s) [upload=12571]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:46:01.440197016 (12h6m1.029542003s) [upload=12571]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:14.557737130 (12h8m14.147082107s) [upload=12571]: Done uploading segments... Aug 7 21:48:14.557758951 (12h8m14.147103938s) [upload=12571]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.531365201s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 33 out of 60 is failed (threshold=3) [..X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...X....................] duration: 1.861217609s FAIL (2025-08-07 21:59:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.861247374s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X................] duration: 6.479408394s FAIL (2025-08-07 21:59:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.479425518s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 29 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXXX..XX..X................] duration: 6.003860861s FAIL (2025-08-07 21:59:37 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.003884076s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 32 out of 60 is failed (threshold=3) [.........X.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..X...............] duration: 1.926785385s FAIL (2025-08-07 21:59:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6664 (11h58m33.293011651s) [upload=12450]: Starting upload Aug 7 21:38:33.703724295 (11h58m33.293069282s) [upload=12450]: Done starting upload Aug 7 21:38:33.703770663 (11h58m33.29311565s) [upload=12450]: Uploading segments... Aug 7 21:38:33.703817212 (11h58m33.293162199s) [upload=12450]: Got next segment. Inline: true Aug 7 21:38:33.703820899 (11h58m33.293165886s) [upload=12450]: Waiting for error group managing segments... Aug 7 21:38:33.703853411 (11h58m33.293198388s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.703857378 (11h58m33.293202365s) [upload=12450]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.703859744 (11h58m33.293204731s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:33.059235349 (11h59m32.648580326s) [upload=12450]: Done uploading segments... Aug 7 21:39:33.059253302 (11h59m32.648598289s) [upload=12450]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:33.956833499 (11h59m33.546178486s) [upload=12464]: Starting upload Aug 7 21:39:33.956907240 (11h59m33.546252227s) [upload=12464]: Done starting upload Aug 7 21:39:33.956967195 (11h59m33.546312182s) [upload=12464]: Uploading segments... Aug 7 21:39:33.957011540 (11h59m33.546356527s) [upload=12464]: Got next segment. Inline: true Aug 7 21:39:33.957016229 (11h59m33.546361226s) [upload=12464]: Waiting for error group managing segments... Aug 7 21:39:33.957053820 (11h59m33.546398807s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:33.957058250 (11h59m33.546403237s) [upload=12464]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:33.957061736 (11h59m33.546406723s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:33.302006486 (12h0m32.891351473s) [upload=12464]: Done uploading segments... Aug 7 21:40:33.302025733 (12h0m32.89137071s) [upload=12464]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:34.211725785 (12h0m33.801070772s) [upload=12482]: Starting upload Aug 7 21:40:34.211806970 (12h0m33.801151957s) [upload=12482]: Done starting upload Aug 7 21:40:34.211965303 (12h0m33.8013103s) [upload=12482]: Uploading segments... Aug 7 21:40:34.212066617 (12h0m33.801411614s) [upload=12482]: Got next segment. Inline: true Aug 7 21:40:34.212072819 (12h0m33.801417806s) [upload=12482]: Waiting for error group managing segments... Aug 7 21:40:34.212126081 (12h0m33.801471068s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:34.212132573 (12h0m33.80147756s) [upload=12482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:34.212136120 (12h0m33.801481107s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:33.557664583 (12h1m33.14700957s) [upload=12482]: Done uploading segments... Aug 7 21:41:33.557694410 (12h1m33.147039387s) [upload=12482]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.468736652 (12h1m34.058081639s) [upload=12501]: Starting upload Aug 7 21:41:34.468803340 (12h1m34.058148327s) [upload=12501]: Done starting upload Aug 7 21:41:34.468862643 (12h1m34.05820763s) [upload=12501]: Uploading segments... Aug 7 21:41:34.468901758 (12h1m34.058246735s) [upload=12501]: Got next segment. Inline: true Aug 7 21:41:34.468905907 (12h1m34.058250884s) [upload=12501]: Waiting for error group managing segments... Aug 7 21:41:34.468943147 (12h1m34.058288134s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.468947105 (12h1m34.058292092s) [upload=12501]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.468949690 (12h1m34.058294677s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:33.814395640 (12h2m33.403740627s) [upload=12501]: Done uploading segments... Aug 7 21:42:33.814423744 (12h2m33.403768731s) [upload=12501]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:46.083099596 (12h2m45.672444573s) [upload=12522]: Starting upload Aug 7 21:42:46.083172004 (12h2m45.672516981s) [upload=12522]: Done starting upload Aug 7 21:42:46.083231107 (12h2m45.672576094s) [upload=12522]: Uploading segments... Aug 7 21:42:46.083339093 (12h2m45.67268408s) [upload=12522]: Got next segment. Inline: true Aug 7 21:42:46.083344093 (12h2m45.67268908s) [upload=12522]: Waiting for error group managing segments... Aug 7 21:42:46.083408827 (12h2m45.672753814s) [upload=12522]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:46.083414187 (12h2m45.672759174s) [upload=12522]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:46.083416732 (12h2m45.672761719s) [upload=12522]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:47.411975663 (12h2m47.00132066s) [upload=12522]: Done uploading segments... Aug 7 21:42:47.411996032 (12h2m47.001341019s) [upload=12522]: Upload finished. err: <nil> Aug 7 21:43:11.548517821 (12h3m11.137862808s) [upload=12529]: Starting upload Aug 7 21:43:11.548568578 (12h3m11.137913565s) [upload=12529]: Done starting upload Aug 7 21:43:11.548617301 (12h3m11.137962288s) [upload=12529]: Uploading segments... Aug 7 21:43:11.548646878 (12h3m11.137991865s) [upload=12529]: Got next segment. Inline: true Aug 7 21:43:11.548650374 (12h3m11.137995362s) [upload=12529]: Waiting for error group managing segments... Aug 7 21:43:11.548681634 (12h3m11.138026621s) [upload=12529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:11.548685361 (12h3m11.138030338s) [upload=12529]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:11.548687726 (12h3m11.138032713s) [upload=12529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:10.906282559 (12h4m10.495627546s) [upload=12529]: Done uploading segments... Aug 7 21:44:10.906303038 (12h4m10.495648015s) [upload=12529]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:44:11.778510580 (12h4m11.367855567s) [upload=12548]: Starting upload Aug 7 21:44:11.778570013 (12h4m11.367915s) [upload=12548]: Done starting upload Aug 7 21:44:11.778662139 (12h4m11.368007126s) [upload=12548]: Uploading segments... Aug 7 21:44:11.778800213 (12h4m11.36814519s) [upload=12548]: Got next segment. Inline: true Aug 7 21:44:11.778804892 (12h4m11.368149869s) [upload=12548]: Waiting for error group managing segments... Aug 7 21:44:11.778861480 (12h4m11.368206467s) [upload=12548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:11.778866230 (12h4m11.368211207s) [upload=12548]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:11.778868745 (12h4m11.368213732s) [upload=12548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:11.131015060 (12h5m10.720360047s) [upload=12548]: Done uploading segments... Aug 7 21:45:11.131033725 (12h5m10.720378712s) [upload=12548]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:45:11.998653609 (12h5m11.587998596s) [upload=12563]: Starting upload Aug 7 21:45:11.998732921 (12h5m11.588077908s) [upload=12563]: Done starting upload Aug 7 21:45:11.998811802 (12h5m11.588156789s) [upload=12563]: Uploading segments... Aug 7 21:45:11.998922443 (12h5m11.58826743s) [upload=12563]: Got next segment. Inline: true Aug 7 21:45:11.998926992 (12h5m11.588271979s) [upload=12563]: Waiting for error group managing segments... Aug 7 21:45:11.998979913 (12h5m11.5883249s) [upload=12563]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:11.998987267 (12h5m11.588332254s) [upload=12563]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:11.998990603 (12h5m11.58833559s) [upload=12563]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:14.137150454 (12h5m13.726495441s) [upload=12563]: Done uploading segments... Aug 7 21:45:14.137168529 (12h5m13.726513516s) [upload=12563]: Upload finished. err: <nil> Aug 7 21:45:53.851988781 (12h5m53.441333768s) [upload=12569]: Starting upload Aug 7 21:45:53.852043686 (12h5m53.441388673s) [upload=12569]: Done starting upload Aug 7 21:45:53.852088772 (12h5m53.441433759s) [upload=12569]: Uploading segments... Aug 7 21:45:53.852141613 (12h5m53.4414866s) [upload=12569]: Got next segment. Inline: true Aug 7 21:45:53.852146192 (12h5m53.441491169s) [upload=12569]: Waiting for error group managing segments... Aug 7 21:45:53.852182612 (12h5m53.441527599s) [upload=12569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:53.852186920 (12h5m53.441531907s) [upload=12569]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:53.852189195 (12h5m53.441534182s) [upload=12569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:54.127817304 (12h5m53.717162291s) [upload=12569]: Done uploading segments... Aug 7 21:45:54.127834497 (12h5m53.717179484s) [upload=12569]: Upload finished. err: <nil> Aug 7 21:46:54.936546775 (12h6m54.525891762s) [upload=12585]: Starting upload Aug 7 21:46:54.936595597 (12h6m54.525940585s) [upload=12585]: Done starting upload Aug 7 21:46:54.936645253 (12h6m54.52599024s) [upload=12585]: Uploading segments... Aug 7 21:46:54.936675961 (12h6m54.526020948s) [upload=12585]: Got next segment. Inline: true Aug 7 21:46:54.936680381 (12h6m54.526025358s) [upload=12585]: Waiting for error group managing segments... Aug 7 21:46:54.936716089 (12h6m54.526061076s) [upload=12585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:54.936720797 (12h6m54.526065774s) [upload=12585]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:46:54.936723172 (12h6m54.526068159s) [upload=12585]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:54.288569811 (12h7m53.877914798s) [upload=12585]: Done uploading segments... Aug 7 21:47:54.288620218 (12h7m53.877965205s) [upload=12585]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.926809771s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X......X..X.............X..X....................X......] duration: 6.484370217s FAIL (2025-08-07 21:59:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 57 (12h15m32.758009944s) [noise=true , address=104.36.82.54:4166 , piece_id=5SBUSGFVYPNFGILW6OTNLGJQG7CNI7GYYEJQJMITPKFZ2MAK5QRQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.168945643 (12h15m32.75829063s) [noise=true , address=104.36.82.12:4179 , piece_id=KFVQX32X6I7QBU552L4NSPZG3JTVT24YESONPWGM6UCTR4ZWBUKQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.168966093 (12h15m32.75831109s) [noise=true , address=104.36.82.43:4850 , piece_id=JH2ASD4NQAQMOBOZMDGFKVPPOS2P26QIFSYDNHYZN24RP52E4XLA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169201393 (12h15m32.75854638s) [noise=true , address=104.36.82.36:4633 , piece_id=6RL3INCLYJCEH53JOFJDL2MMEXGQ2G4T33PKZPJ3W2FG5QMCFLCQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169453466 (12h15m32.758798453s) [noise=true , address=104.36.82.47:4520 , piece_id=K75KQHBYH7QTZDCFM7ZW55EUO2W3UN2NEIKUMNCMQWHYYNOWLL3A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169588975 (12h15m32.758933962s) [noise=true , address=104.36.82.9:4014 , piece_id=D3NLTEDBRB4M7GIMPYCDNQ66HYLAZGSSBRM7SJWU45LPTCRMYMCQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170078472 (12h15m32.759423459s) [noise=true , address=104.36.82.39:4739 , piece_id=QGKQ3NJO7OTN5O2QQU43KYJXARJW6JQMSCONEJR35AEBBNRORJCA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170313160 (12h15m32.759658147s) [noise=true , address=104.36.82.62:4429 , piece_id=RCPKZHWWS56QSEVG6L25B3FMQOU5ZES3BNAEYIUYQ3PLQV3FDMVQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170515587 (12h15m32.759860574s) [noise=true , address=79.127.201.215:4353 , piece_id=II3JPNVAATS6APC4JGPPOJRHQRMHS5EYGLLEEAMTOPHPEGRQUF7A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170753382 (12h15m32.760098359s) [noise=true , address=45.91.116.8:4065 , piece_id=OEVAVKCHFYMJMGBKUKF5F34O3NH47DG45QJHHNWMOD7TDRMCQLAQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170965789 (12h15m32.760310776s) [noise=true , address=77.90.31.14:4678 , piece_id=7WZVWEAFKI4ZEIHTZOJNMAQWLBVPPLB7U4JNYZQNXCVRC2S4BUBA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171167525 (12h15m32.760512512s) [noise=true , address=104.36.82.45:4926 , piece_id=S3EH75AGZ3QVGR6RM3RI3SCHXZJPE7LORX2QQA5SCEADV3RDYM6A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171354372 (12h15m32.760699359s) [noise=true , address=64.49.13.154:4935 , piece_id=O6TZCBWBHVESLRX76EFTUT4U4OIKFFVXO5UKLMWAJA2BDTHIHN2A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171604611 (12h15m32.760949598s) [noise=true , address=104.36.82.44:4894 , piece_id=WCLKE2VTORDUBR5II5IQ5OTYMNHCMP2OYAIBHG3GV5P5U3FCO7HQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171788884 (12h15m32.761133861s) [noise=true , address=104.36.82.37:4674 , piece_id=AN3RCVXXXNEJUD33D4W47FDFYJQ7X5ZK7H4BJD6NR4PJOJSCVV6A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.181382449 (12h15m32.770727426s) [noise=true , address=104.36.82.23:4474 , piece_id=WM2SVCDZEFAN6VXQ2PWKBZGUCHIHNJ52RNKJKUNBMJ3AHFXUT3PA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:34.281100029 (12h15m33.870445006s) [upload=12734]: Next returned nil segment Aug 7 21:55:34.281113294 (12h15m33.870458281s) [upload=12734]: Waiting for error group managing segments... Aug 7 21:55:34.581166654 (12h15m34.170511631s) [noise=true , address=64.49.13.154:4935 , piece_id=O6TZCBWBHVESLRX76EFTUT4U4OIKFFVXO5UKLMWAJA2BDTHIHN2A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.075145374 (12h15m34.664490361s) [noise=true , address=79.127.201.215:4353 , piece_id=II3JPNVAATS6APC4JGPPOJRHQRMHS5EYGLLEEAMTOPHPEGRQUF7A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.117523204 (12h15m34.706868191s) [noise=true , address=77.90.31.14:4678 , piece_id=7WZVWEAFKI4ZEIHTZOJNMAQWLBVPPLB7U4JNYZQNXCVRC2S4BUBA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.189259324 (12h15m34.778604311s) [noise=true , address=104.36.82.9:4014 , piece_id=D3NLTEDBRB4M7GIMPYCDNQ66HYLAZGSSBRM7SJWU45LPTCRMYMCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.208589096 (12h15m34.797934083s) [noise=true , address=104.36.82.23:4474 , piece_id=WM2SVCDZEFAN6VXQ2PWKBZGUCHIHNJ52RNKJKUNBMJ3AHFXUT3PA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.240233681 (12h15m34.829578668s) [noise=true , address=104.36.82.58:4265 , piece_id=XMDZZK372B4DNLPJRKD2BM7OI7JNQSZ6ABLSXNPZEH2WT73AT7TA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.249509658 (12h15m34.838854645s) [noise=true , address=104.36.82.39:4739 , piece_id=QGKQ3NJO7OTN5O2QQU43KYJXARJW6JQMSCONEJR35AEBBNRORJCA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.254602933 (12h15m34.84394792s) [noise=true , address=104.36.82.12:4179 , piece_id=KFVQX32X6I7QBU552L4NSPZG3JTVT24YESONPWGM6UCTR4ZWBUKQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.255644015 (12h15m34.844989002s) [noise=true , address=104.36.82.59:4322 , piece_id=O6L7GOHWQOA6VAMNJATA232B7BEQS66XKSZXUAQLX3KYNQXTMUCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.260038844 (12h15m34.849383831s) [noise=true , address=45.91.116.8:4065 , piece_id=OEVAVKCHFYMJMGBKUKF5F34O3NH47DG45QJHHNWMOD7TDRMCQLAQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.160031235 (12h15m35.749376222s) [noise=true , address=104.36.82.43:4850 , piece_id=JH2ASD4NQAQMOBOZMDGFKVPPOS2P26QIFSYDNHYZN24RP52E4XLA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.204255979 (12h15m35.793600956s) [noise=true , address=104.36.82.62:4429 , piece_id=RCPKZHWWS56QSEVG6L25B3FMQOU5ZES3BNAEYIUYQ3PLQV3FDMVQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.608134744 (12h15m36.197479731s) [noise=true , address=104.36.82.37:4674 , piece_id=AN3RCVXXXNEJUD33D4W47FDFYJQ7X5ZK7H4BJD6NR4PJOJSCVV6A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.985735083 (12h15m36.57508007s) [noise=true , address=104.36.82.45:4926 , piece_id=S3EH75AGZ3QVGR6RM3RI3SCHXZJPE7LORX2QQA5SCEADV3RDYM6A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.985758668 (12h15m36.575103655s) [seg_pos={0 0}, upload=12734]: Segment reached optimal threshold of 14 pieces. Aug 7 21:55:36.985891332 (12h15m36.575236319s) [noise=true , address=104.36.82.36:4633 , piece_id=6RL3INCLYJCEH53JOFJDL2MMEXGQ2G4T33PKZPJ3W2FG5QMCFLCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dvBPvb9YEtM4NieVCXL71Pak5zwiMg14MSXH4RuLZts4dByG8): protocol: expected piece hash; context canceled; EOF Aug 7 21:55:36.986091084 (12h15m36.575436071s) [noise=true , address=104.36.82.47:4520 , piece_id=K75KQHBYH7QTZDCFM7ZW55EUO2W3UN2NEIKUMNCMQWHYYNOWLL3A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy): protocol: write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection; write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection Aug 7 21:55:36.986139326 (12h15m36.575484333s) [noise=true , address=104.36.82.54:4166 , piece_id=5SBUSGFVYPNFGILW6OTNLGJQG7CNI7GYYEJQJMITPKFZ2MAK5QRQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv): protocol: expected piece hash; context canceled; EOF Aug 7 21:55:36.986268794 (12h15m36.575613791s) [noise=true , address=104.36.82.44:4894 , piece_id=WCLKE2VTORDUBR5II5IQ5OTYMNHCMP2OYAIBHG3GV5P5U3FCO7HQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44): protocol: read tcp 188.166.225.170:34542->104.36.82.44:4894: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:55:36.986293281 (12h15m36.575638268s) [seg_pos={0 0}, upload=12734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:55:36.986385397 (12h15m36.575730394s) [upload=12734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:55:36.986391038 (12h15m36.575736025s) [upload=12734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:55:37.437633211 (12h15m37.026978238s) [upload=12734]: Done uploading segments... Aug 7 21:55:37.437656937 (12h15m37.027001924s) [upload=12734]: Upload finished. err: <nil> Aug 7 21:55:48.800457325 (12h15m48.389802332s) [upload=12741]: Starting upload Aug 7 21:55:48.800589529 (12h15m48.389934526s) [upload=12741]: Done starting upload Aug 7 21:55:48.800688879 (12h15m48.390033866s) [upload=12741]: Uploading segments... Aug 7 21:55:48.801167825 (12h15m48.390512802s) [upload=12741]: Got next segment. Inline: false Aug 7 21:55:48.801200477 (12h15m48.390545464s) [upload=12741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:55:48.801204775 (12h15m48.390549762s) [upload=12741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:55:49.015107648 (12h15m48.604452635s) [upload=12741]: Next segment err: metaclient: Too Many Requests Aug 7 21:55:49.015124520 (12h15m48.604469507s) [upload=12741]: Done uploading segments... Aug 7 21:55:49.015136393 (12h15m48.60448138s) [upload=12741]: Upload finished. err: metaclient: Too Many Requests Duration: 6.484384153s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.............X....X.X...X...................................] duration: 747.034521ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:59:29 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 747.057696ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 33 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X...X.....................] duration: 1.850472957s FAIL (2025-08-07 21:59:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== line exceeded Aug 7 21:37:22.572699237 (11h57m22.162044224s) [upload=12432]: Starting upload Aug 7 21:37:22.572782937 (11h57m22.162127924s) [upload=12432]: Done starting upload Aug 7 21:37:22.572842260 (11h57m22.162187247s) [upload=12432]: Uploading segments... Aug 7 21:37:22.572885313 (11h57m22.1622303s) [upload=12432]: Got next segment. Inline: true Aug 7 21:37:22.572889591 (11h57m22.162234578s) [upload=12432]: Waiting for error group managing segments... Aug 7 21:37:22.572930138 (11h57m22.162275125s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:22.572934296 (11h57m22.162279283s) [upload=12432]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:22.572937121 (11h57m22.162282118s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.933954276 (11h57m51.523299253s) [upload=12432]: Done uploading segments... Aug 7 21:37:51.933970927 (11h57m51.523315914s) [upload=12432]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:28.745940745 (11h58m28.335285732s) [upload=12448]: Starting upload Aug 7 21:38:28.745997504 (11h58m28.335342491s) [upload=12448]: Done starting upload Aug 7 21:38:28.746040176 (11h58m28.335385163s) [upload=12448]: Uploading segments... Aug 7 21:38:28.746078458 (11h58m28.335423446s) [upload=12448]: Got next segment. Inline: true Aug 7 21:38:28.746081766 (11h58m28.335426743s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:38:28.746111923 (11h58m28.33545691s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:28.746115540 (11h58m28.335460527s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:28.746118205 (11h58m28.335463192s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:58.105315693 (11h58m57.694660691s) [upload=12448]: Done uploading segments... Aug 7 21:38:58.105342465 (11h58m57.694687452s) [upload=12448]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:27.137238449 (11h59m26.726583436s) [upload=12462]: Starting upload Aug 7 21:39:27.137294907 (11h59m26.726639894s) [upload=12462]: Done starting upload Aug 7 21:39:27.137337368 (11h59m26.726682355s) [upload=12462]: Uploading segments... Aug 7 21:39:27.137366043 (11h59m26.72671103s) [upload=12462]: Got next segment. Inline: true Aug 7 21:39:27.137370061 (11h59m26.726715048s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:39:27.137401901 (11h59m26.726746879s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:27.137405518 (11h59m26.726750495s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:27.137407813 (11h59m26.72675279s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.495186447 (11h59m56.084531434s) [upload=12462]: Done uploading segments... Aug 7 21:39:56.495206415 (11h59m56.084551402s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:01.914615953 (12h0m1.50396094s) [upload=12475]: Starting upload Aug 7 21:40:01.914684295 (12h0m1.504029282s) [upload=12475]: Done starting upload Aug 7 21:40:01.914741163 (12h0m1.50408616s) [upload=12475]: Uploading segments... Aug 7 21:40:01.914783103 (12h0m1.50412809s) [upload=12475]: Got next segment. Inline: true Aug 7 21:40:01.914788143 (12h0m1.504133131s) [upload=12475]: Waiting for error group managing segments... Aug 7 21:40:01.914833950 (12h0m1.504178937s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:01.914840073 (12h0m1.50418507s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:01.914846014 (12h0m1.504191011s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:31.268196986 (12h0m30.857542023s) [upload=12475]: Done uploading segments... Aug 7 21:40:31.268223536 (12h0m30.857568533s) [upload=12475]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:46.248438399 (12h0m45.837783386s) [upload=12487]: Starting upload Aug 7 21:40:46.248491410 (12h0m45.837836387s) [upload=12487]: Done starting upload Aug 7 21:40:46.248540414 (12h0m45.837885401s) [upload=12487]: Uploading segments... Aug 7 21:40:46.248572345 (12h0m45.837917332s) [upload=12487]: Got next segment. Inline: true Aug 7 21:40:46.248576133 (12h0m45.83792112s) [upload=12487]: Waiting for error group managing segments... Aug 7 21:40:46.248613013 (12h0m45.837958s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:46.248616850 (12h0m45.837961837s) [upload=12487]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:46.248619255 (12h0m45.837964242s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:55.261300379 (12h0m54.850645366s) [upload=12487]: Done uploading segments... Aug 7 21:40:55.261318514 (12h0m54.850663501s) [upload=12487]: Upload finished. err: <nil> Aug 7 21:42:00.373248983 (12h1m59.962594s) [upload=12509]: Starting upload Aug 7 21:42:00.373363813 (12h1m59.96270884s) [upload=12509]: Done starting upload Aug 7 21:42:00.373438045 (12h1m59.962783062s) [upload=12509]: Uploading segments... Aug 7 21:42:00.373501075 (12h1m59.962846083s) [upload=12509]: Got next segment. Inline: true Aug 7 21:42:00.373507809 (12h1m59.962852816s) [upload=12509]: Waiting for error group managing segments... Aug 7 21:42:00.373577642 (12h1m59.962922669s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:00.373584325 (12h1m59.962929342s) [upload=12509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:00.373590016 (12h1m59.962935053s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:29.724044552 (12h2m29.313389539s) [upload=12509]: Done uploading segments... Aug 7 21:42:29.724072415 (12h2m29.313417392s) [upload=12509]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:30.611135457 (12h2m30.200480444s) [upload=12516]: Starting upload Aug 7 21:42:30.611201894 (12h2m30.200546881s) [upload=12516]: Done starting upload Aug 7 21:42:30.611252732 (12h2m30.200597719s) [upload=12516]: Uploading segments... Aug 7 21:42:30.611338045 (12h2m30.200683032s) [upload=12516]: Got next segment. Inline: true Aug 7 21:42:30.611344076 (12h2m30.200689063s) [upload=12516]: Waiting for error group managing segments... Aug 7 21:42:30.611400083 (12h2m30.20074507s) [upload=12516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:30.611404922 (12h2m30.200749909s) [upload=12516]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:30.611408178 (12h2m30.200753166s) [upload=12516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:46.271341803 (12h2m45.860686781s) [upload=12516]: Done uploading segments... Aug 7 21:42:46.271358525 (12h2m45.860703512s) [upload=12516]: Upload finished. err: <nil> Aug 7 21:43:05.160958738 (12h3m4.750303725s) [upload=12528]: Starting upload Aug 7 21:43:05.161064639 (12h3m4.750409626s) [upload=12528]: Done starting upload Aug 7 21:43:05.161177936 (12h3m4.750522953s) [upload=12528]: Uploading segments... Aug 7 21:43:05.161261357 (12h3m4.750606344s) [upload=12528]: Got next segment. Inline: true Aug 7 21:43:05.161266376 (12h3m4.750611353s) [upload=12528]: Waiting for error group managing segments... Aug 7 21:43:05.161306081 (12h3m4.750651069s) [upload=12528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:05.161310580 (12h3m4.750655567s) [upload=12528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:05.161312864 (12h3m4.750657841s) [upload=12528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:17.899724589 (12h3m17.489069576s) [upload=12528]: Done uploading segments... Aug 7 21:43:17.899747483 (12h3m17.48909251s) [upload=12528]: Upload finished. err: <nil> Aug 7 21:43:34.894582431 (12h3m34.483927418s) [upload=12537]: Starting upload Aug 7 21:43:34.894647576 (12h3m34.483992564s) [upload=12537]: Done starting upload Aug 7 21:43:34.894699496 (12h3m34.484044483s) [upload=12537]: Uploading segments... Aug 7 21:43:34.894732940 (12h3m34.484077927s) [upload=12537]: Got next segment. Inline: true Aug 7 21:43:34.894736967 (12h3m34.484081954s) [upload=12537]: Waiting for error group managing segments... Aug 7 21:43:34.894770662 (12h3m34.484115649s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:34.894775050 (12h3m34.484120027s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:34.894777795 (12h3m34.484122782s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:35.191263901 (12h3m34.780608888s) [upload=12537]: Done uploading segments... Aug 7 21:43:35.191280001 (12h3m34.780624978s) [upload=12537]: Upload finished. err: <nil> Aug 7 21:44:27.756622626 (12h4m27.345967613s) [upload=12553]: Starting upload Aug 7 21:44:27.756679294 (12h4m27.346024281s) [upload=12553]: Done starting upload Aug 7 21:44:27.756721205 (12h4m27.346066192s) [upload=12553]: Uploading segments... Aug 7 21:44:27.756751423 (12h4m27.34609641s) [upload=12553]: Got next segment. Inline: true Aug 7 21:44:27.756754779 (12h4m27.346099766s) [upload=12553]: Waiting for error group managing segments... Aug 7 21:44:27.756785678 (12h4m27.346130655s) [upload=12553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:27.756789615 (12h4m27.346134602s) [upload=12553]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:27.756791790 (12h4m27.346136777s) [upload=12553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:57.108386877 (12h4m56.697731864s) [upload=12553]: Done uploading segments... Aug 7 21:44:57.108405372 (12h4m56.697750359s) [upload=12553]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.850520086s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [.......................XX.XX..X.XX..........................] duration: 231.331688ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:59:23 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 231.358809ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 29 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXXX..XX..X...............] duration: 5.995710988s FAIL (2025-08-07 21:59:19 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.995738991s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [......X...............X..........X.................X........] duration: 2.273646461s FAIL (2025-08-07 21:59:15 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 2.27367285s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 34 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.........................] duration: 2.567858966s FAIL (2025-08-07 21:59:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.567876168s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 33 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X...X....................] duration: 1.845921291s FAIL (2025-08-07 21:59:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== line exceeded Aug 7 21:37:22.572699237 (11h57m22.162044224s) [upload=12432]: Starting upload Aug 7 21:37:22.572782937 (11h57m22.162127924s) [upload=12432]: Done starting upload Aug 7 21:37:22.572842260 (11h57m22.162187247s) [upload=12432]: Uploading segments... Aug 7 21:37:22.572885313 (11h57m22.1622303s) [upload=12432]: Got next segment. Inline: true Aug 7 21:37:22.572889591 (11h57m22.162234578s) [upload=12432]: Waiting for error group managing segments... Aug 7 21:37:22.572930138 (11h57m22.162275125s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:22.572934296 (11h57m22.162279283s) [upload=12432]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:22.572937121 (11h57m22.162282118s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.933954276 (11h57m51.523299253s) [upload=12432]: Done uploading segments... Aug 7 21:37:51.933970927 (11h57m51.523315914s) [upload=12432]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:28.745940745 (11h58m28.335285732s) [upload=12448]: Starting upload Aug 7 21:38:28.745997504 (11h58m28.335342491s) [upload=12448]: Done starting upload Aug 7 21:38:28.746040176 (11h58m28.335385163s) [upload=12448]: Uploading segments... Aug 7 21:38:28.746078458 (11h58m28.335423446s) [upload=12448]: Got next segment. Inline: true Aug 7 21:38:28.746081766 (11h58m28.335426743s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:38:28.746111923 (11h58m28.33545691s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:28.746115540 (11h58m28.335460527s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:28.746118205 (11h58m28.335463192s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:58.105315693 (11h58m57.694660691s) [upload=12448]: Done uploading segments... Aug 7 21:38:58.105342465 (11h58m57.694687452s) [upload=12448]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:27.137238449 (11h59m26.726583436s) [upload=12462]: Starting upload Aug 7 21:39:27.137294907 (11h59m26.726639894s) [upload=12462]: Done starting upload Aug 7 21:39:27.137337368 (11h59m26.726682355s) [upload=12462]: Uploading segments... Aug 7 21:39:27.137366043 (11h59m26.72671103s) [upload=12462]: Got next segment. Inline: true Aug 7 21:39:27.137370061 (11h59m26.726715048s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:39:27.137401901 (11h59m26.726746879s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:27.137405518 (11h59m26.726750495s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:27.137407813 (11h59m26.72675279s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.495186447 (11h59m56.084531434s) [upload=12462]: Done uploading segments... Aug 7 21:39:56.495206415 (11h59m56.084551402s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:01.914615953 (12h0m1.50396094s) [upload=12475]: Starting upload Aug 7 21:40:01.914684295 (12h0m1.504029282s) [upload=12475]: Done starting upload Aug 7 21:40:01.914741163 (12h0m1.50408616s) [upload=12475]: Uploading segments... Aug 7 21:40:01.914783103 (12h0m1.50412809s) [upload=12475]: Got next segment. Inline: true Aug 7 21:40:01.914788143 (12h0m1.504133131s) [upload=12475]: Waiting for error group managing segments... Aug 7 21:40:01.914833950 (12h0m1.504178937s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:01.914840073 (12h0m1.50418507s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:01.914846014 (12h0m1.504191011s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:31.268196986 (12h0m30.857542023s) [upload=12475]: Done uploading segments... Aug 7 21:40:31.268223536 (12h0m30.857568533s) [upload=12475]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:46.248438399 (12h0m45.837783386s) [upload=12487]: Starting upload Aug 7 21:40:46.248491410 (12h0m45.837836387s) [upload=12487]: Done starting upload Aug 7 21:40:46.248540414 (12h0m45.837885401s) [upload=12487]: Uploading segments... Aug 7 21:40:46.248572345 (12h0m45.837917332s) [upload=12487]: Got next segment. Inline: true Aug 7 21:40:46.248576133 (12h0m45.83792112s) [upload=12487]: Waiting for error group managing segments... Aug 7 21:40:46.248613013 (12h0m45.837958s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:46.248616850 (12h0m45.837961837s) [upload=12487]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:46.248619255 (12h0m45.837964242s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:55.261300379 (12h0m54.850645366s) [upload=12487]: Done uploading segments... Aug 7 21:40:55.261318514 (12h0m54.850663501s) [upload=12487]: Upload finished. err: <nil> Aug 7 21:42:00.373248983 (12h1m59.962594s) [upload=12509]: Starting upload Aug 7 21:42:00.373363813 (12h1m59.96270884s) [upload=12509]: Done starting upload Aug 7 21:42:00.373438045 (12h1m59.962783062s) [upload=12509]: Uploading segments... Aug 7 21:42:00.373501075 (12h1m59.962846083s) [upload=12509]: Got next segment. Inline: true Aug 7 21:42:00.373507809 (12h1m59.962852816s) [upload=12509]: Waiting for error group managing segments... Aug 7 21:42:00.373577642 (12h1m59.962922669s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:00.373584325 (12h1m59.962929342s) [upload=12509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:00.373590016 (12h1m59.962935053s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:29.724044552 (12h2m29.313389539s) [upload=12509]: Done uploading segments... Aug 7 21:42:29.724072415 (12h2m29.313417392s) [upload=12509]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:30.611135457 (12h2m30.200480444s) [upload=12516]: Starting upload Aug 7 21:42:30.611201894 (12h2m30.200546881s) [upload=12516]: Done starting upload Aug 7 21:42:30.611252732 (12h2m30.200597719s) [upload=12516]: Uploading segments... Aug 7 21:42:30.611338045 (12h2m30.200683032s) [upload=12516]: Got next segment. Inline: true Aug 7 21:42:30.611344076 (12h2m30.200689063s) [upload=12516]: Waiting for error group managing segments... Aug 7 21:42:30.611400083 (12h2m30.20074507s) [upload=12516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:30.611404922 (12h2m30.200749909s) [upload=12516]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:30.611408178 (12h2m30.200753166s) [upload=12516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:46.271341803 (12h2m45.860686781s) [upload=12516]: Done uploading segments... Aug 7 21:42:46.271358525 (12h2m45.860703512s) [upload=12516]: Upload finished. err: <nil> Aug 7 21:43:05.160958738 (12h3m4.750303725s) [upload=12528]: Starting upload Aug 7 21:43:05.161064639 (12h3m4.750409626s) [upload=12528]: Done starting upload Aug 7 21:43:05.161177936 (12h3m4.750522953s) [upload=12528]: Uploading segments... Aug 7 21:43:05.161261357 (12h3m4.750606344s) [upload=12528]: Got next segment. Inline: true Aug 7 21:43:05.161266376 (12h3m4.750611353s) [upload=12528]: Waiting for error group managing segments... Aug 7 21:43:05.161306081 (12h3m4.750651069s) [upload=12528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:05.161310580 (12h3m4.750655567s) [upload=12528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:05.161312864 (12h3m4.750657841s) [upload=12528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:17.899724589 (12h3m17.489069576s) [upload=12528]: Done uploading segments... Aug 7 21:43:17.899747483 (12h3m17.48909251s) [upload=12528]: Upload finished. err: <nil> Aug 7 21:43:34.894582431 (12h3m34.483927418s) [upload=12537]: Starting upload Aug 7 21:43:34.894647576 (12h3m34.483992564s) [upload=12537]: Done starting upload Aug 7 21:43:34.894699496 (12h3m34.484044483s) [upload=12537]: Uploading segments... Aug 7 21:43:34.894732940 (12h3m34.484077927s) [upload=12537]: Got next segment. Inline: true Aug 7 21:43:34.894736967 (12h3m34.484081954s) [upload=12537]: Waiting for error group managing segments... Aug 7 21:43:34.894770662 (12h3m34.484115649s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:34.894775050 (12h3m34.484120027s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:34.894777795 (12h3m34.484122782s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:35.191263901 (12h3m34.780608888s) [upload=12537]: Done uploading segments... Aug 7 21:43:35.191280001 (12h3m34.780624978s) [upload=12537]: Upload finished. err: <nil> Aug 7 21:44:27.756622626 (12h4m27.345967613s) [upload=12553]: Starting upload Aug 7 21:44:27.756679294 (12h4m27.346024281s) [upload=12553]: Done starting upload Aug 7 21:44:27.756721205 (12h4m27.346066192s) [upload=12553]: Uploading segments... Aug 7 21:44:27.756751423 (12h4m27.34609641s) [upload=12553]: Got next segment. Inline: true Aug 7 21:44:27.756754779 (12h4m27.346099766s) [upload=12553]: Waiting for error group managing segments... Aug 7 21:44:27.756785678 (12h4m27.346130655s) [upload=12553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:27.756789615 (12h4m27.346134602s) [upload=12553]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:27.756791790 (12h4m27.346136777s) [upload=12553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:57.108386877 (12h4m56.697731864s) [upload=12553]: Done uploading segments... Aug 7 21:44:57.108405372 (12h4m56.697750359s) [upload=12553]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.845943674s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 26 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXXXX.....X................] duration: 7.491884229s FAIL (2025-08-07 21:58:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== t segment. Inline: false Aug 7 21:44:25.061589945 (12h4m24.650934922s) [upload=12552]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:25.061594724 (12h4m24.650939711s) [upload=12552]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:44:50.602885060 (12h4m50.192230047s) [seg_pos={0 0}, upload=12552]: Begin upload segment... Aug 7 21:44:50.602935446 (12h4m50.192280423s) [seg_pos={0 0}, upload=12552]: Done begin upload segment. Aug 7 21:44:50.603112565 (12h4m50.192457552s) [noise=true , address=79.127.205.233:4624 , piece_id=WEAYHGF7VE6W5ML3QAE3AITMEJGQFH4JQZ7RQKR7O4NKTPR6RZRA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.603695610 (12h4m50.193040597s) [noise=true , address=104.36.82.22:4529 , piece_id=ZTTBM5H6MHVMQ3OFVTVISN6WJ7WB3XBWI7YZKUXCFNQ6NO3D4CGA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.604300768 (12h4m50.193645755s) [noise=true , address=45.91.116.6:4002 , piece_id=AE5EL7NWD6FEQ7NXX2JRHB4MVZQDXRGQU3EER2X7COISFAX6P3XQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.604588519 (12h4m50.193933496s) [noise=true , address=104.36.82.56:4249 , piece_id=RD4B7IDIJV25L7EJNSYCZGW5QKJEM435E2N3UFUD44Q4S3DCBOXQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605153860 (12h4m50.194498837s) [noise=true , address=77.90.31.11:4562 , piece_id=XC6F2WLSXKQB7E7MBRNZBJKUSUFL6XNA5TVX5M3ZJI3ZZMK7UBLA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605455377 (12h4m50.194800364s) [noise=true , address=104.36.82.23:4466 , piece_id=5J5X2LSP6GCX2PNRIDZYFJPISWSX3YSRID5M2WU6Y5Z6LWRHLC4Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605743810 (12h4m50.195088797s) [noise=true , address=104.36.82.62:4420 , piece_id=UOO36PYVE5WFRRZLGYLJ6I6SV4EO7W2ZH2TIXJ2OIZZMY3LTHR2A, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.606547457 (12h4m50.195892444s) [noise=true , address=104.36.82.5:4842 , piece_id=E5DSJXN63RFX7EQ4ZTBMUESFW77EB3HQNA5KCQGRVRAA5XO2GN3Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.606982099 (12h4m50.196327086s) [noise=true , address=104.36.82.59:4301 , piece_id=VVR5CQWCPNPZUFEFCPB6ZL2KASWS5XO65FVDI474RKRZSJFDUQTA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.607295849 (12h4m50.196640836s) [noise=true , address=45.91.116.14:4198 , piece_id=2HRU3JZDQSNGD5NJZDLZSRG2WIO2Y2QBSOAY6KUEGUKDBWM53RSA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.607695593 (12h4m50.19704058s) [noise=true , address=104.36.82.6:4881 , piece_id=VUC33D7TLVFICDLA3334DEKBJE5E5P4HQTJU3VDQNMVYWVESSJWA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.608364103 (12h4m50.19770909s) [noise=true , address=104.36.82.2:4757 , piece_id=H6KNYP5HASPWQIHADTMEZDC5AE5HITCAY73XGNWVEXJOPDEZ3SHA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.608764288 (12h4m50.198109265s) [noise=true , address=104.36.82.46:4493 , piece_id=PS4S4OZ6HSITEJFPJDIBWQ5RGWTDRKMX3NCEWXCLC2PSP5Q2KPZQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609065284 (12h4m50.198410271s) [noise=true , address=104.36.82.47:4518 , piece_id=KJ3PJCT3EDH3WZBT7AWWKI6ABGVVAMKZTVJY7Q76FULC72TUWA6Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609312777 (12h4m50.198657755s) [noise=true , address=104.36.82.60:4349 , piece_id=GTTF32YMJWBY2GZEFZPIBZXZWNI7MBLUYCTQ3SRFG735PLWXTXBQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609589888 (12h4m50.198934875s) [noise=true , address=104.36.82.52:4116 , piece_id=7QA2Q5ACN43TQUOKWA34EZFSSQBE3OWNJTYODHHQ5GCVOA4QZOJA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.610295969 (12h4m50.199641056s) [noise=true , address=104.36.82.37:4678 , piece_id=VJO5JO3SJEXBQZC6PJ34DRKWEQKOPI33ZF4IHNNOOTSD7IBRR4PQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.610811305 (12h4m50.200156302s) [noise=true , address=64.49.13.169:4714 , piece_id=A6O3H7QLE6KQTHWUKOEROVHPEHVLG22KNF2KCV3HK2KCLXP7SYVA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:51.715421583 (12h4m51.30476657s) [upload=12552]: Next returned nil segment Aug 7 21:44:51.715438125 (12h4m51.304783112s) [upload=12552]: Waiting for error group managing segments... Aug 7 21:44:51.998131281 (12h4m51.587476268s) [noise=true , address=64.49.13.169:4714 , piece_id=A6O3H7QLE6KQTHWUKOEROVHPEHVLG22KNF2KCV3HK2KCLXP7SYVA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:51.999236335 (12h4m51.588581312s) [noise=true , address=79.127.205.233:4624 , piece_id=WEAYHGF7VE6W5ML3QAE3AITMEJGQFH4JQZ7RQKR7O4NKTPR6RZRA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.638299991 (12h4m52.227644978s) [noise=true , address=45.91.116.6:4002 , piece_id=AE5EL7NWD6FEQ7NXX2JRHB4MVZQDXRGQU3EER2X7COISFAX6P3XQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.639927565 (12h4m52.229272542s) [noise=true , address=104.36.82.6:4881 , piece_id=VUC33D7TLVFICDLA3334DEKBJE5E5P4HQTJU3VDQNMVYWVESSJWA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.643220585 (12h4m52.232565572s) [noise=true , address=104.36.82.2:4757 , piece_id=H6KNYP5HASPWQIHADTMEZDC5AE5HITCAY73XGNWVEXJOPDEZ3SHA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.711872926 (12h4m52.301217913s) [noise=true , address=104.36.82.56:4249 , piece_id=RD4B7IDIJV25L7EJNSYCZGW5QKJEM435E2N3UFUD44Q4S3DCBOXQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.763601896 (12h4m52.352946883s) [noise=true , address=104.36.82.22:4529 , piece_id=ZTTBM5H6MHVMQ3OFVTVISN6WJ7WB3XBWI7YZKUXCFNQ6NO3D4CGA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.768238797 (12h4m52.357583785s) [noise=true , address=104.36.82.60:4349 , piece_id=GTTF32YMJWBY2GZEFZPIBZXZWNI7MBLUYCTQ3SRFG735PLWXTXBQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.798543408 (12h4m52.387888395s) [noise=true , address=45.91.116.14:4198 , piece_id=2HRU3JZDQSNGD5NJZDLZSRG2WIO2Y2QBSOAY6KUEGUKDBWM53RSA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.861192704 (12h4m52.450537691s) [noise=true , address=77.90.31.11:4562 , piece_id=XC6F2WLSXKQB7E7MBRNZBJKUSUFL6XNA5TVX5M3ZJI3ZZMK7UBLA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:53.070218164 (12h4m52.659563151s) [noise=true , address=104.36.82.59:4301 , piece_id=VVR5CQWCPNPZUFEFCPB6ZL2KASWS5XO65FVDI474RKRZSJFDUQTA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:53.558930346 (12h4m53.148275333s) [noise=true , address=104.36.82.46:4493 , piece_id=PS4S4OZ6HSITEJFPJDIBWQ5RGWTDRKMX3NCEWXCLC2PSP5Q2KPZQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.132342777 (12h4m54.721687765s) [noise=true , address=104.36.82.37:4678 , piece_id=VJO5JO3SJEXBQZC6PJ34DRKWEQKOPI33ZF4IHNNOOTSD7IBRR4PQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.265236558 (12h4m54.854581545s) [noise=true , address=104.36.82.52:4116 , piece_id=7QA2Q5ACN43TQUOKWA34EZFSSQBE3OWNJTYODHHQ5GCVOA4QZOJA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.265254161 (12h4m54.854599138s) [seg_pos={0 0}, upload=12552]: Segment reached optimal threshold of 14 pieces. Aug 7 21:44:55.265482308 (12h4m54.854827295s) [noise=true , address=104.36.82.5:4842 , piece_id=E5DSJXN63RFX7EQ4ZTBMUESFW77EB3HQNA5KCQGRVRAA5XO2GN3Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn): protocol: expected piece hash; write tcp 188.166.225.170:57020->104.36.82.5:4842: use of closed network connection; write tcp 188.166.225.170:57020->104.36.82.5:4842: use of closed network connection; EOF Aug 7 21:44:55.265551430 (12h4m54.854896417s) [noise=true , address=104.36.82.47:4518 , piece_id=KJ3PJCT3EDH3WZBT7AWWKI6ABGVVAMKZTVJY7Q76FULC72TUWA6Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: expected piece hash; context canceled; EOF Aug 7 21:44:55.265766391 (12h4m54.855111378s) [noise=true , address=104.36.82.23:4466 , piece_id=5J5X2LSP6GCX2PNRIDZYFJPISWSX3YSRID5M2WU6Y5Z6LWRHLC4Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rkNnk9RUhKBJ3EsakqMRb3wYwrhRLhB4ZYbrdfKPDk8erRJpP): protocol: write tcp 188.166.225.170:49724->104.36.82.23:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:44:55.265824172 (12h4m54.855169159s) [noise=true , address=104.36.82.62:4420 , piece_id=UOO36PYVE5WFRRZLGYLJ6I6SV4EO7W2ZH2TIXJ2OIZZMY3LTHR2A, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SWVQwL5PxDLaSXADwTrAYS2hVcNBuYAA6ReWuRM3UZa57QyYi): protocol: read tcp 188.166.225.170:33092->104.36.82.62:4420: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:44:55.265833430 (12h4m54.855178417s) [seg_pos={0 0}, upload=12552]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:44:55.265888796 (12h4m54.855233783s) [upload=12552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:44:55.265892303 (12h4m54.85523728s) [upload=12552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:46.088275738 (12h8m45.677620725s) [upload=12552]: Done uploading segments... Aug 7 21:48:46.088301307 (12h8m45.677646284s) [upload=12552]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.491907573s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X...............] duration: 6.225380814s FAIL (2025-08-07 21:58:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.225401705s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXXX...X..X................] duration: 5.534265102s FAIL (2025-08-07 21:58:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.534295831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [..X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.X.X...............] duration: 7.241997161s FAIL (2025-08-07 21:58:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:46242->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:46242->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46242->104.36.82.11:4115: use of closed network connection Aug 7 21:44:51.028127350 (12h4m50.617472327s) [seg_pos={0 0}, upload=12555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:44:51.028226048 (12h4m50.617571035s) [upload=12555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:44:51.028237751 (12h4m50.617582738s) [upload=12555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:51.322864709 (12h4m50.912209697s) [upload=12555]: Done uploading segments... Aug 7 21:44:51.322882523 (12h4m50.9122275s) [upload=12555]: Upload finished. err: <nil> Aug 7 21:45:58.815791269 (12h5m58.405136256s) [upload=12571]: Starting upload Aug 7 21:45:58.815846996 (12h5m58.405191983s) [upload=12571]: Done starting upload Aug 7 21:45:58.815953540 (12h5m58.405298527s) [upload=12571]: Uploading segments... Aug 7 21:45:58.816065333 (12h5m58.405410321s) [upload=12571]: Got next segment. Inline: false Aug 7 21:45:58.816100661 (12h5m58.405445638s) [upload=12571]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:58.816106212 (12h5m58.405451199s) [upload=12571]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:45:59.064845605 (12h5m58.654190592s) [seg_pos={0 0}, upload=12571]: Begin upload segment... Aug 7 21:45:59.064885521 (12h5m58.654230508s) [seg_pos={0 0}, upload=12571]: Done begin upload segment. Aug 7 21:45:59.064910399 (12h5m58.654255376s) [noise=true , address=77.90.31.18:4807 , piece_id=63E5KXGYEJHGHYE4SKTNTMED5R4HSWIST4CR4EV6ZGON3YHV73AA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.064952560 (12h5m58.654297547s) [noise=true , address=45.91.116.12:4227 , piece_id=MDQGJ65RT4QEUFX6OFXPEAOFSTG2VXBRC23DXVVD2IV6V52UFF4Q, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065245920 (12h5m58.654590907s) [noise=true , address=45.91.116.8:4048 , piece_id=2LZRN76O3QS6FODVI52DIIMK7Z77QQBGIXYUAKZCSFI7WMGRTDUA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065360070 (12h5m58.654705057s) [noise=true , address=45.91.116.9:4262 , piece_id=D4ABUUAX4MHYLFPR6LN2REDC6VRT2W7D4VZHHYM7GZ6HWOWMG2LA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065517019 (12h5m58.654861996s) [noise=true , address=77.90.31.19:4843 , piece_id=S73TOGLK5HDCBE4BINZ72A3URDR6TUD43IVIFDY5V37EVCN6222A, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065615889 (12h5m58.654960876s) [noise=true , address=104.36.82.6:4875 , piece_id=W43EX62262JONZAGLUREOG3EGMICRBIVX5O445J3OQDQ2O475J7A, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065778009 (12h5m58.655122986s) [noise=true , address=104.36.82.11:4115 , piece_id=AH7EPSWFJFX7XDYUQOFCNXEHKQ5AMQX22NY6OFBMMHATCOVH4YJA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065884293 (12h5m58.65522928s) [noise=true , address=45.91.116.8:4047 , piece_id=TEMT2SH64IMIPQ3GDE5XCF2R2XLY33HUKRFF6KUWEYH7CDVYD4YQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066024761 (12h5m58.655369748s) [noise=true , address=77.90.31.11:4554 , piece_id=LMSB62TMAJONFULH4DAOTPWQMGLUFRZYKVG4HNJ4AUBP7EBP5FFQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066243109 (12h5m58.655588096s) [noise=true , address=45.91.116.6:4973 , piece_id=7MZSPAEXC3Q7W7U3QMZUQRUI3OHPQI3XRAJW4OGU7UZ73FEI4LIQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066452850 (12h5m58.655797837s) [noise=true , address=104.36.82.6:4874 , piece_id=T3W6RCAW4YPEU42CESOSUSFSNPDBFXR74WZUYQ2AHJNBZV776MVQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066669685 (12h5m58.656014672s) [noise=true , address=45.91.116.6:4975 , piece_id=CI7SM56L5KPNQEV52PLIETEBQVH5CFXTKVQOQIZ73YOPEGYC3KOA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066955041 (12h5m58.656300028s) [noise=true , address=104.36.82.7:4909 , piece_id=EKLBHF6P5BXX5VLSLL4QEOJXQXWM7MCOY67W5OON2UHRNMKOQFAA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.067149393 (12h5m58.65649438s) [noise=true , address=45.91.116.13:4154 , piece_id=LDSWQ62RRGHTT544S6463WLDKW3WFBWIFQGQDHLYWJJD2ESMYJNQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.067334066 (12h5m58.656679053s) [noise=true , address=104.36.82.16:4294 , piece_id=KDCYWDEXSTKCHR23CAOKKTYUMRQS5Y4LH6KIJPQFF737WA5TST7Q, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.067527215 (12h5m58.656872202s) [noise=true , address=77.90.31.15:4698 , piece_id=6L3IAVPA4PQGSMKOGSKWYG4NLM6TSA2YSIAM6S6MYSDUW26CP5CA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:46:00.855806043 (12h6m0.44515103s) [upload=12571]: Next returned nil segment Aug 7 21:46:00.855821061 (12h6m0.445166048s) [upload=12571]: Waiting for error group managing segments... Aug 7 21:46:01.180369497 (12h6m0.769714484s) [noise=true , address=77.90.31.15:4698 , piece_id=6L3IAVPA4PQGSMKOGSKWYG4NLM6TSA2YSIAM6S6MYSDUW26CP5CA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.200492297 (12h6m0.789837284s) [noise=true , address=45.91.116.6:4975 , piece_id=CI7SM56L5KPNQEV52PLIETEBQVH5CFXTKVQOQIZ73YOPEGYC3KOA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.210262691 (12h6m0.799607678s) [noise=true , address=104.36.82.16:4294 , piece_id=KDCYWDEXSTKCHR23CAOKKTYUMRQS5Y4LH6KIJPQFF737WA5TST7Q, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.220625478 (12h6m0.809970465s) [noise=true , address=104.36.82.7:4909 , piece_id=EKLBHF6P5BXX5VLSLL4QEOJXQXWM7MCOY67W5OON2UHRNMKOQFAA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.304447281 (12h6m0.893792308s) [noise=true , address=104.36.82.11:4115 , piece_id=AH7EPSWFJFX7XDYUQOFCNXEHKQ5AMQX22NY6OFBMMHATCOVH4YJA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.374462398 (12h6m0.963807385s) [noise=true , address=45.91.116.8:4047 , piece_id=TEMT2SH64IMIPQ3GDE5XCF2R2XLY33HUKRFF6KUWEYH7CDVYD4YQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.378351098 (12h6m0.967696085s) [noise=true , address=104.36.82.6:4875 , piece_id=W43EX62262JONZAGLUREOG3EGMICRBIVX5O445J3OQDQ2O475J7A, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.395257154 (12h6m0.984602141s) [noise=true , address=45.91.116.6:4973 , piece_id=7MZSPAEXC3Q7W7U3QMZUQRUI3OHPQI3XRAJW4OGU7UZ73FEI4LIQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.404312138 (12h6m0.993657125s) [noise=true , address=77.90.31.19:4843 , piece_id=S73TOGLK5HDCBE4BINZ72A3URDR6TUD43IVIFDY5V37EVCN6222A, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.409522908 (12h6m0.998867885s) [noise=true , address=45.91.116.13:4154 , piece_id=LDSWQ62RRGHTT544S6463WLDKW3WFBWIFQGQDHLYWJJD2ESMYJNQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.416315134 (12h6m1.005660121s) [noise=true , address=45.91.116.12:4227 , piece_id=MDQGJ65RT4QEUFX6OFXPEAOFSTG2VXBRC23DXVVD2IV6V52UFF4Q, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.421432725 (12h6m1.010777712s) [noise=true , address=45.91.116.8:4048 , piece_id=2LZRN76O3QS6FODVI52DIIMK7Z77QQBGIXYUAKZCSFI7WMGRTDUA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.439709112 (12h6m1.029054099s) [noise=true , address=45.91.116.9:4262 , piece_id=D4ABUUAX4MHYLFPR6LN2REDC6VRT2W7D4VZHHYM7GZ6HWOWMG2LA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.439735472 (12h6m1.029080459s) [seg_pos={0 0}, upload=12571]: Segment reached optimal threshold of 13 pieces. Aug 7 21:46:01.440013234 (12h6m1.029358221s) [noise=true , address=77.90.31.11:4554 , piece_id=LMSB62TMAJONFULH4DAOTPWQMGLUFRZYKVG4HNJ4AUBP7EBP5FFQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:01.440020518 (12h6m1.029365505s) [noise=true , address=104.36.82.6:4874 , piece_id=T3W6RCAW4YPEU42CESOSUSFSNPDBFXR74WZUYQ2AHJNBZV776MVQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: write tcp 188.166.225.170:54068->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:01.440109398 (12h6m1.029454385s) [noise=true , address=77.90.31.18:4807 , piece_id=63E5KXGYEJHGHYE4SKTNTMED5R4HSWIST4CR4EV6ZGON3YHV73AA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:01.440122312 (12h6m1.029467299s) [seg_pos={0 0}, upload=12571]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:46:01.440193058 (12h6m1.029538045s) [upload=12571]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:46:01.440197016 (12h6m1.029542003s) [upload=12571]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:14.557737130 (12h8m14.147082107s) [upload=12571]: Done uploading segments... Aug 7 21:48:14.557758951 (12h8m14.147103938s) [upload=12571]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.242021859s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [..............X....X.X...X..................................] duration: 859.898862ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:58:38 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 859.936505ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 33 out of 60 is failed (threshold=3) [...X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...X...................] duration: 1.885910891s FAIL (2025-08-07 21:58:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.885948113s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X......X..X.............X..X....................X.....] duration: 6.859009995s FAIL (2025-08-07 21:58:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 57 (12h15m32.758009944s) [noise=true , address=104.36.82.54:4166 , piece_id=5SBUSGFVYPNFGILW6OTNLGJQG7CNI7GYYEJQJMITPKFZ2MAK5QRQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.168945643 (12h15m32.75829063s) [noise=true , address=104.36.82.12:4179 , piece_id=KFVQX32X6I7QBU552L4NSPZG3JTVT24YESONPWGM6UCTR4ZWBUKQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.168966093 (12h15m32.75831109s) [noise=true , address=104.36.82.43:4850 , piece_id=JH2ASD4NQAQMOBOZMDGFKVPPOS2P26QIFSYDNHYZN24RP52E4XLA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169201393 (12h15m32.75854638s) [noise=true , address=104.36.82.36:4633 , piece_id=6RL3INCLYJCEH53JOFJDL2MMEXGQ2G4T33PKZPJ3W2FG5QMCFLCQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169453466 (12h15m32.758798453s) [noise=true , address=104.36.82.47:4520 , piece_id=K75KQHBYH7QTZDCFM7ZW55EUO2W3UN2NEIKUMNCMQWHYYNOWLL3A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169588975 (12h15m32.758933962s) [noise=true , address=104.36.82.9:4014 , piece_id=D3NLTEDBRB4M7GIMPYCDNQ66HYLAZGSSBRM7SJWU45LPTCRMYMCQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170078472 (12h15m32.759423459s) [noise=true , address=104.36.82.39:4739 , piece_id=QGKQ3NJO7OTN5O2QQU43KYJXARJW6JQMSCONEJR35AEBBNRORJCA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170313160 (12h15m32.759658147s) [noise=true , address=104.36.82.62:4429 , piece_id=RCPKZHWWS56QSEVG6L25B3FMQOU5ZES3BNAEYIUYQ3PLQV3FDMVQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170515587 (12h15m32.759860574s) [noise=true , address=79.127.201.215:4353 , piece_id=II3JPNVAATS6APC4JGPPOJRHQRMHS5EYGLLEEAMTOPHPEGRQUF7A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170753382 (12h15m32.760098359s) [noise=true , address=45.91.116.8:4065 , piece_id=OEVAVKCHFYMJMGBKUKF5F34O3NH47DG45QJHHNWMOD7TDRMCQLAQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170965789 (12h15m32.760310776s) [noise=true , address=77.90.31.14:4678 , piece_id=7WZVWEAFKI4ZEIHTZOJNMAQWLBVPPLB7U4JNYZQNXCVRC2S4BUBA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171167525 (12h15m32.760512512s) [noise=true , address=104.36.82.45:4926 , piece_id=S3EH75AGZ3QVGR6RM3RI3SCHXZJPE7LORX2QQA5SCEADV3RDYM6A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171354372 (12h15m32.760699359s) [noise=true , address=64.49.13.154:4935 , piece_id=O6TZCBWBHVESLRX76EFTUT4U4OIKFFVXO5UKLMWAJA2BDTHIHN2A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171604611 (12h15m32.760949598s) [noise=true , address=104.36.82.44:4894 , piece_id=WCLKE2VTORDUBR5II5IQ5OTYMNHCMP2OYAIBHG3GV5P5U3FCO7HQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171788884 (12h15m32.761133861s) [noise=true , address=104.36.82.37:4674 , piece_id=AN3RCVXXXNEJUD33D4W47FDFYJQ7X5ZK7H4BJD6NR4PJOJSCVV6A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.181382449 (12h15m32.770727426s) [noise=true , address=104.36.82.23:4474 , piece_id=WM2SVCDZEFAN6VXQ2PWKBZGUCHIHNJ52RNKJKUNBMJ3AHFXUT3PA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:34.281100029 (12h15m33.870445006s) [upload=12734]: Next returned nil segment Aug 7 21:55:34.281113294 (12h15m33.870458281s) [upload=12734]: Waiting for error group managing segments... Aug 7 21:55:34.581166654 (12h15m34.170511631s) [noise=true , address=64.49.13.154:4935 , piece_id=O6TZCBWBHVESLRX76EFTUT4U4OIKFFVXO5UKLMWAJA2BDTHIHN2A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.075145374 (12h15m34.664490361s) [noise=true , address=79.127.201.215:4353 , piece_id=II3JPNVAATS6APC4JGPPOJRHQRMHS5EYGLLEEAMTOPHPEGRQUF7A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.117523204 (12h15m34.706868191s) [noise=true , address=77.90.31.14:4678 , piece_id=7WZVWEAFKI4ZEIHTZOJNMAQWLBVPPLB7U4JNYZQNXCVRC2S4BUBA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.189259324 (12h15m34.778604311s) [noise=true , address=104.36.82.9:4014 , piece_id=D3NLTEDBRB4M7GIMPYCDNQ66HYLAZGSSBRM7SJWU45LPTCRMYMCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.208589096 (12h15m34.797934083s) [noise=true , address=104.36.82.23:4474 , piece_id=WM2SVCDZEFAN6VXQ2PWKBZGUCHIHNJ52RNKJKUNBMJ3AHFXUT3PA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.240233681 (12h15m34.829578668s) [noise=true , address=104.36.82.58:4265 , piece_id=XMDZZK372B4DNLPJRKD2BM7OI7JNQSZ6ABLSXNPZEH2WT73AT7TA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.249509658 (12h15m34.838854645s) [noise=true , address=104.36.82.39:4739 , piece_id=QGKQ3NJO7OTN5O2QQU43KYJXARJW6JQMSCONEJR35AEBBNRORJCA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.254602933 (12h15m34.84394792s) [noise=true , address=104.36.82.12:4179 , piece_id=KFVQX32X6I7QBU552L4NSPZG3JTVT24YESONPWGM6UCTR4ZWBUKQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.255644015 (12h15m34.844989002s) [noise=true , address=104.36.82.59:4322 , piece_id=O6L7GOHWQOA6VAMNJATA232B7BEQS66XKSZXUAQLX3KYNQXTMUCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.260038844 (12h15m34.849383831s) [noise=true , address=45.91.116.8:4065 , piece_id=OEVAVKCHFYMJMGBKUKF5F34O3NH47DG45QJHHNWMOD7TDRMCQLAQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.160031235 (12h15m35.749376222s) [noise=true , address=104.36.82.43:4850 , piece_id=JH2ASD4NQAQMOBOZMDGFKVPPOS2P26QIFSYDNHYZN24RP52E4XLA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.204255979 (12h15m35.793600956s) [noise=true , address=104.36.82.62:4429 , piece_id=RCPKZHWWS56QSEVG6L25B3FMQOU5ZES3BNAEYIUYQ3PLQV3FDMVQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.608134744 (12h15m36.197479731s) [noise=true , address=104.36.82.37:4674 , piece_id=AN3RCVXXXNEJUD33D4W47FDFYJQ7X5ZK7H4BJD6NR4PJOJSCVV6A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.985735083 (12h15m36.57508007s) [noise=true , address=104.36.82.45:4926 , piece_id=S3EH75AGZ3QVGR6RM3RI3SCHXZJPE7LORX2QQA5SCEADV3RDYM6A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.985758668 (12h15m36.575103655s) [seg_pos={0 0}, upload=12734]: Segment reached optimal threshold of 14 pieces. Aug 7 21:55:36.985891332 (12h15m36.575236319s) [noise=true , address=104.36.82.36:4633 , piece_id=6RL3INCLYJCEH53JOFJDL2MMEXGQ2G4T33PKZPJ3W2FG5QMCFLCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dvBPvb9YEtM4NieVCXL71Pak5zwiMg14MSXH4RuLZts4dByG8): protocol: expected piece hash; context canceled; EOF Aug 7 21:55:36.986091084 (12h15m36.575436071s) [noise=true , address=104.36.82.47:4520 , piece_id=K75KQHBYH7QTZDCFM7ZW55EUO2W3UN2NEIKUMNCMQWHYYNOWLL3A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy): protocol: write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection; write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection Aug 7 21:55:36.986139326 (12h15m36.575484333s) [noise=true , address=104.36.82.54:4166 , piece_id=5SBUSGFVYPNFGILW6OTNLGJQG7CNI7GYYEJQJMITPKFZ2MAK5QRQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv): protocol: expected piece hash; context canceled; EOF Aug 7 21:55:36.986268794 (12h15m36.575613791s) [noise=true , address=104.36.82.44:4894 , piece_id=WCLKE2VTORDUBR5II5IQ5OTYMNHCMP2OYAIBHG3GV5P5U3FCO7HQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44): protocol: read tcp 188.166.225.170:34542->104.36.82.44:4894: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:55:36.986293281 (12h15m36.575638268s) [seg_pos={0 0}, upload=12734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:55:36.986385397 (12h15m36.575730394s) [upload=12734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:55:36.986391038 (12h15m36.575736025s) [upload=12734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:55:37.437633211 (12h15m37.026978238s) [upload=12734]: Done uploading segments... Aug 7 21:55:37.437656937 (12h15m37.027001924s) [upload=12734]: Upload finished. err: <nil> Aug 7 21:55:48.800457325 (12h15m48.389802332s) [upload=12741]: Starting upload Aug 7 21:55:48.800589529 (12h15m48.389934526s) [upload=12741]: Done starting upload Aug 7 21:55:48.800688879 (12h15m48.390033866s) [upload=12741]: Uploading segments... Aug 7 21:55:48.801167825 (12h15m48.390512802s) [upload=12741]: Got next segment. Inline: false Aug 7 21:55:48.801200477 (12h15m48.390545464s) [upload=12741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:55:48.801204775 (12h15m48.390549762s) [upload=12741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:55:49.015107648 (12h15m48.604452635s) [upload=12741]: Next segment err: metaclient: Too Many Requests Aug 7 21:55:49.015124520 (12h15m48.604469507s) [upload=12741]: Done uploading segments... Aug 7 21:55:49.015136393 (12h15m48.60448138s) [upload=12741]: Upload finished. err: metaclient: Too Many Requests Duration: 6.859035144s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 32 out of 60 is failed (threshold=3) [..........X.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..X..............] duration: 1.90519502s FAIL (2025-08-07 21:58:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6664 (11h58m33.293011651s) [upload=12450]: Starting upload Aug 7 21:38:33.703724295 (11h58m33.293069282s) [upload=12450]: Done starting upload Aug 7 21:38:33.703770663 (11h58m33.29311565s) [upload=12450]: Uploading segments... Aug 7 21:38:33.703817212 (11h58m33.293162199s) [upload=12450]: Got next segment. Inline: true Aug 7 21:38:33.703820899 (11h58m33.293165886s) [upload=12450]: Waiting for error group managing segments... Aug 7 21:38:33.703853411 (11h58m33.293198388s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.703857378 (11h58m33.293202365s) [upload=12450]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.703859744 (11h58m33.293204731s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:33.059235349 (11h59m32.648580326s) [upload=12450]: Done uploading segments... Aug 7 21:39:33.059253302 (11h59m32.648598289s) [upload=12450]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:33.956833499 (11h59m33.546178486s) [upload=12464]: Starting upload Aug 7 21:39:33.956907240 (11h59m33.546252227s) [upload=12464]: Done starting upload Aug 7 21:39:33.956967195 (11h59m33.546312182s) [upload=12464]: Uploading segments... Aug 7 21:39:33.957011540 (11h59m33.546356527s) [upload=12464]: Got next segment. Inline: true Aug 7 21:39:33.957016229 (11h59m33.546361226s) [upload=12464]: Waiting for error group managing segments... Aug 7 21:39:33.957053820 (11h59m33.546398807s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:33.957058250 (11h59m33.546403237s) [upload=12464]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:33.957061736 (11h59m33.546406723s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:33.302006486 (12h0m32.891351473s) [upload=12464]: Done uploading segments... Aug 7 21:40:33.302025733 (12h0m32.89137071s) [upload=12464]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:34.211725785 (12h0m33.801070772s) [upload=12482]: Starting upload Aug 7 21:40:34.211806970 (12h0m33.801151957s) [upload=12482]: Done starting upload Aug 7 21:40:34.211965303 (12h0m33.8013103s) [upload=12482]: Uploading segments... Aug 7 21:40:34.212066617 (12h0m33.801411614s) [upload=12482]: Got next segment. Inline: true Aug 7 21:40:34.212072819 (12h0m33.801417806s) [upload=12482]: Waiting for error group managing segments... Aug 7 21:40:34.212126081 (12h0m33.801471068s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:34.212132573 (12h0m33.80147756s) [upload=12482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:34.212136120 (12h0m33.801481107s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:33.557664583 (12h1m33.14700957s) [upload=12482]: Done uploading segments... Aug 7 21:41:33.557694410 (12h1m33.147039387s) [upload=12482]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.468736652 (12h1m34.058081639s) [upload=12501]: Starting upload Aug 7 21:41:34.468803340 (12h1m34.058148327s) [upload=12501]: Done starting upload Aug 7 21:41:34.468862643 (12h1m34.05820763s) [upload=12501]: Uploading segments... Aug 7 21:41:34.468901758 (12h1m34.058246735s) [upload=12501]: Got next segment. Inline: true Aug 7 21:41:34.468905907 (12h1m34.058250884s) [upload=12501]: Waiting for error group managing segments... Aug 7 21:41:34.468943147 (12h1m34.058288134s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.468947105 (12h1m34.058292092s) [upload=12501]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.468949690 (12h1m34.058294677s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:33.814395640 (12h2m33.403740627s) [upload=12501]: Done uploading segments... Aug 7 21:42:33.814423744 (12h2m33.403768731s) [upload=12501]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:46.083099596 (12h2m45.672444573s) [upload=12522]: Starting upload Aug 7 21:42:46.083172004 (12h2m45.672516981s) [upload=12522]: Done starting upload Aug 7 21:42:46.083231107 (12h2m45.672576094s) [upload=12522]: Uploading segments... Aug 7 21:42:46.083339093 (12h2m45.67268408s) [upload=12522]: Got next segment. Inline: true Aug 7 21:42:46.083344093 (12h2m45.67268908s) [upload=12522]: Waiting for error group managing segments... Aug 7 21:42:46.083408827 (12h2m45.672753814s) [upload=12522]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:46.083414187 (12h2m45.672759174s) [upload=12522]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:46.083416732 (12h2m45.672761719s) [upload=12522]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:47.411975663 (12h2m47.00132066s) [upload=12522]: Done uploading segments... Aug 7 21:42:47.411996032 (12h2m47.001341019s) [upload=12522]: Upload finished. err: <nil> Aug 7 21:43:11.548517821 (12h3m11.137862808s) [upload=12529]: Starting upload Aug 7 21:43:11.548568578 (12h3m11.137913565s) [upload=12529]: Done starting upload Aug 7 21:43:11.548617301 (12h3m11.137962288s) [upload=12529]: Uploading segments... Aug 7 21:43:11.548646878 (12h3m11.137991865s) [upload=12529]: Got next segment. Inline: true Aug 7 21:43:11.548650374 (12h3m11.137995362s) [upload=12529]: Waiting for error group managing segments... Aug 7 21:43:11.548681634 (12h3m11.138026621s) [upload=12529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:11.548685361 (12h3m11.138030338s) [upload=12529]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:11.548687726 (12h3m11.138032713s) [upload=12529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:10.906282559 (12h4m10.495627546s) [upload=12529]: Done uploading segments... Aug 7 21:44:10.906303038 (12h4m10.495648015s) [upload=12529]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:44:11.778510580 (12h4m11.367855567s) [upload=12548]: Starting upload Aug 7 21:44:11.778570013 (12h4m11.367915s) [upload=12548]: Done starting upload Aug 7 21:44:11.778662139 (12h4m11.368007126s) [upload=12548]: Uploading segments... Aug 7 21:44:11.778800213 (12h4m11.36814519s) [upload=12548]: Got next segment. Inline: true Aug 7 21:44:11.778804892 (12h4m11.368149869s) [upload=12548]: Waiting for error group managing segments... Aug 7 21:44:11.778861480 (12h4m11.368206467s) [upload=12548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:11.778866230 (12h4m11.368211207s) [upload=12548]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:11.778868745 (12h4m11.368213732s) [upload=12548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:11.131015060 (12h5m10.720360047s) [upload=12548]: Done uploading segments... Aug 7 21:45:11.131033725 (12h5m10.720378712s) [upload=12548]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:45:11.998653609 (12h5m11.587998596s) [upload=12563]: Starting upload Aug 7 21:45:11.998732921 (12h5m11.588077908s) [upload=12563]: Done starting upload Aug 7 21:45:11.998811802 (12h5m11.588156789s) [upload=12563]: Uploading segments... Aug 7 21:45:11.998922443 (12h5m11.58826743s) [upload=12563]: Got next segment. Inline: true Aug 7 21:45:11.998926992 (12h5m11.588271979s) [upload=12563]: Waiting for error group managing segments... Aug 7 21:45:11.998979913 (12h5m11.5883249s) [upload=12563]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:11.998987267 (12h5m11.588332254s) [upload=12563]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:11.998990603 (12h5m11.58833559s) [upload=12563]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:14.137150454 (12h5m13.726495441s) [upload=12563]: Done uploading segments... Aug 7 21:45:14.137168529 (12h5m13.726513516s) [upload=12563]: Upload finished. err: <nil> Aug 7 21:45:53.851988781 (12h5m53.441333768s) [upload=12569]: Starting upload Aug 7 21:45:53.852043686 (12h5m53.441388673s) [upload=12569]: Done starting upload Aug 7 21:45:53.852088772 (12h5m53.441433759s) [upload=12569]: Uploading segments... Aug 7 21:45:53.852141613 (12h5m53.4414866s) [upload=12569]: Got next segment. Inline: true Aug 7 21:45:53.852146192 (12h5m53.441491169s) [upload=12569]: Waiting for error group managing segments... Aug 7 21:45:53.852182612 (12h5m53.441527599s) [upload=12569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:53.852186920 (12h5m53.441531907s) [upload=12569]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:53.852189195 (12h5m53.441534182s) [upload=12569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:54.127817304 (12h5m53.717162291s) [upload=12569]: Done uploading segments... Aug 7 21:45:54.127834497 (12h5m53.717179484s) [upload=12569]: Upload finished. err: <nil> Aug 7 21:46:54.936546775 (12h6m54.525891762s) [upload=12585]: Starting upload Aug 7 21:46:54.936595597 (12h6m54.525940585s) [upload=12585]: Done starting upload Aug 7 21:46:54.936645253 (12h6m54.52599024s) [upload=12585]: Uploading segments... Aug 7 21:46:54.936675961 (12h6m54.526020948s) [upload=12585]: Got next segment. Inline: true Aug 7 21:46:54.936680381 (12h6m54.526025358s) [upload=12585]: Waiting for error group managing segments... Aug 7 21:46:54.936716089 (12h6m54.526061076s) [upload=12585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:54.936720797 (12h6m54.526065774s) [upload=12585]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:46:54.936723172 (12h6m54.526068159s) [upload=12585]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:54.288569811 (12h7m53.877914798s) [upload=12585]: Done uploading segments... Aug 7 21:47:54.288620218 (12h7m53.877965205s) [upload=12585]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.905246246s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [........................XX.XX..X.XX.........................] duration: 239.962891ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:58:25 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 240.000474ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [...X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.X.X..............] duration: 6.146706711s FAIL (2025-08-07 21:58:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:46242->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:46242->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46242->104.36.82.11:4115: use of closed network connection Aug 7 21:44:51.028127350 (12h4m50.617472327s) [seg_pos={0 0}, upload=12555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:44:51.028226048 (12h4m50.617571035s) [upload=12555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:44:51.028237751 (12h4m50.617582738s) [upload=12555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:51.322864709 (12h4m50.912209697s) [upload=12555]: Done uploading segments... Aug 7 21:44:51.322882523 (12h4m50.9122275s) [upload=12555]: Upload finished. err: <nil> Aug 7 21:45:58.815791269 (12h5m58.405136256s) [upload=12571]: Starting upload Aug 7 21:45:58.815846996 (12h5m58.405191983s) [upload=12571]: Done starting upload Aug 7 21:45:58.815953540 (12h5m58.405298527s) [upload=12571]: Uploading segments... Aug 7 21:45:58.816065333 (12h5m58.405410321s) [upload=12571]: Got next segment. Inline: false Aug 7 21:45:58.816100661 (12h5m58.405445638s) [upload=12571]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:58.816106212 (12h5m58.405451199s) [upload=12571]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:45:59.064845605 (12h5m58.654190592s) [seg_pos={0 0}, upload=12571]: Begin upload segment... Aug 7 21:45:59.064885521 (12h5m58.654230508s) [seg_pos={0 0}, upload=12571]: Done begin upload segment. Aug 7 21:45:59.064910399 (12h5m58.654255376s) [noise=true , address=77.90.31.18:4807 , piece_id=63E5KXGYEJHGHYE4SKTNTMED5R4HSWIST4CR4EV6ZGON3YHV73AA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.064952560 (12h5m58.654297547s) [noise=true , address=45.91.116.12:4227 , piece_id=MDQGJ65RT4QEUFX6OFXPEAOFSTG2VXBRC23DXVVD2IV6V52UFF4Q, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065245920 (12h5m58.654590907s) [noise=true , address=45.91.116.8:4048 , piece_id=2LZRN76O3QS6FODVI52DIIMK7Z77QQBGIXYUAKZCSFI7WMGRTDUA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065360070 (12h5m58.654705057s) [noise=true , address=45.91.116.9:4262 , piece_id=D4ABUUAX4MHYLFPR6LN2REDC6VRT2W7D4VZHHYM7GZ6HWOWMG2LA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065517019 (12h5m58.654861996s) [noise=true , address=77.90.31.19:4843 , piece_id=S73TOGLK5HDCBE4BINZ72A3URDR6TUD43IVIFDY5V37EVCN6222A, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065615889 (12h5m58.654960876s) [noise=true , address=104.36.82.6:4875 , piece_id=W43EX62262JONZAGLUREOG3EGMICRBIVX5O445J3OQDQ2O475J7A, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065778009 (12h5m58.655122986s) [noise=true , address=104.36.82.11:4115 , piece_id=AH7EPSWFJFX7XDYUQOFCNXEHKQ5AMQX22NY6OFBMMHATCOVH4YJA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065884293 (12h5m58.65522928s) [noise=true , address=45.91.116.8:4047 , piece_id=TEMT2SH64IMIPQ3GDE5XCF2R2XLY33HUKRFF6KUWEYH7CDVYD4YQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066024761 (12h5m58.655369748s) [noise=true , address=77.90.31.11:4554 , piece_id=LMSB62TMAJONFULH4DAOTPWQMGLUFRZYKVG4HNJ4AUBP7EBP5FFQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066243109 (12h5m58.655588096s) [noise=true , address=45.91.116.6:4973 , piece_id=7MZSPAEXC3Q7W7U3QMZUQRUI3OHPQI3XRAJW4OGU7UZ73FEI4LIQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066452850 (12h5m58.655797837s) [noise=true , address=104.36.82.6:4874 , piece_id=T3W6RCAW4YPEU42CESOSUSFSNPDBFXR74WZUYQ2AHJNBZV776MVQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066669685 (12h5m58.656014672s) [noise=true , address=45.91.116.6:4975 , piece_id=CI7SM56L5KPNQEV52PLIETEBQVH5CFXTKVQOQIZ73YOPEGYC3KOA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066955041 (12h5m58.656300028s) [noise=true , address=104.36.82.7:4909 , piece_id=EKLBHF6P5BXX5VLSLL4QEOJXQXWM7MCOY67W5OON2UHRNMKOQFAA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.067149393 (12h5m58.65649438s) [noise=true , address=45.91.116.13:4154 , piece_id=LDSWQ62RRGHTT544S6463WLDKW3WFBWIFQGQDHLYWJJD2ESMYJNQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.067334066 (12h5m58.656679053s) [noise=true , address=104.36.82.16:4294 , piece_id=KDCYWDEXSTKCHR23CAOKKTYUMRQS5Y4LH6KIJPQFF737WA5TST7Q, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.067527215 (12h5m58.656872202s) [noise=true , address=77.90.31.15:4698 , piece_id=6L3IAVPA4PQGSMKOGSKWYG4NLM6TSA2YSIAM6S6MYSDUW26CP5CA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:46:00.855806043 (12h6m0.44515103s) [upload=12571]: Next returned nil segment Aug 7 21:46:00.855821061 (12h6m0.445166048s) [upload=12571]: Waiting for error group managing segments... Aug 7 21:46:01.180369497 (12h6m0.769714484s) [noise=true , address=77.90.31.15:4698 , piece_id=6L3IAVPA4PQGSMKOGSKWYG4NLM6TSA2YSIAM6S6MYSDUW26CP5CA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.200492297 (12h6m0.789837284s) [noise=true , address=45.91.116.6:4975 , piece_id=CI7SM56L5KPNQEV52PLIETEBQVH5CFXTKVQOQIZ73YOPEGYC3KOA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.210262691 (12h6m0.799607678s) [noise=true , address=104.36.82.16:4294 , piece_id=KDCYWDEXSTKCHR23CAOKKTYUMRQS5Y4LH6KIJPQFF737WA5TST7Q, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.220625478 (12h6m0.809970465s) [noise=true , address=104.36.82.7:4909 , piece_id=EKLBHF6P5BXX5VLSLL4QEOJXQXWM7MCOY67W5OON2UHRNMKOQFAA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.304447281 (12h6m0.893792308s) [noise=true , address=104.36.82.11:4115 , piece_id=AH7EPSWFJFX7XDYUQOFCNXEHKQ5AMQX22NY6OFBMMHATCOVH4YJA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.374462398 (12h6m0.963807385s) [noise=true , address=45.91.116.8:4047 , piece_id=TEMT2SH64IMIPQ3GDE5XCF2R2XLY33HUKRFF6KUWEYH7CDVYD4YQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.378351098 (12h6m0.967696085s) [noise=true , address=104.36.82.6:4875 , piece_id=W43EX62262JONZAGLUREOG3EGMICRBIVX5O445J3OQDQ2O475J7A, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.395257154 (12h6m0.984602141s) [noise=true , address=45.91.116.6:4973 , piece_id=7MZSPAEXC3Q7W7U3QMZUQRUI3OHPQI3XRAJW4OGU7UZ73FEI4LIQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.404312138 (12h6m0.993657125s) [noise=true , address=77.90.31.19:4843 , piece_id=S73TOGLK5HDCBE4BINZ72A3URDR6TUD43IVIFDY5V37EVCN6222A, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.409522908 (12h6m0.998867885s) [noise=true , address=45.91.116.13:4154 , piece_id=LDSWQ62RRGHTT544S6463WLDKW3WFBWIFQGQDHLYWJJD2ESMYJNQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.416315134 (12h6m1.005660121s) [noise=true , address=45.91.116.12:4227 , piece_id=MDQGJ65RT4QEUFX6OFXPEAOFSTG2VXBRC23DXVVD2IV6V52UFF4Q, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.421432725 (12h6m1.010777712s) [noise=true , address=45.91.116.8:4048 , piece_id=2LZRN76O3QS6FODVI52DIIMK7Z77QQBGIXYUAKZCSFI7WMGRTDUA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.439709112 (12h6m1.029054099s) [noise=true , address=45.91.116.9:4262 , piece_id=D4ABUUAX4MHYLFPR6LN2REDC6VRT2W7D4VZHHYM7GZ6HWOWMG2LA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.439735472 (12h6m1.029080459s) [seg_pos={0 0}, upload=12571]: Segment reached optimal threshold of 13 pieces. Aug 7 21:46:01.440013234 (12h6m1.029358221s) [noise=true , address=77.90.31.11:4554 , piece_id=LMSB62TMAJONFULH4DAOTPWQMGLUFRZYKVG4HNJ4AUBP7EBP5FFQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:01.440020518 (12h6m1.029365505s) [noise=true , address=104.36.82.6:4874 , piece_id=T3W6RCAW4YPEU42CESOSUSFSNPDBFXR74WZUYQ2AHJNBZV776MVQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: write tcp 188.166.225.170:54068->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:01.440109398 (12h6m1.029454385s) [noise=true , address=77.90.31.18:4807 , piece_id=63E5KXGYEJHGHYE4SKTNTMED5R4HSWIST4CR4EV6ZGON3YHV73AA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:01.440122312 (12h6m1.029467299s) [seg_pos={0 0}, upload=12571]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:46:01.440193058 (12h6m1.029538045s) [upload=12571]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:46:01.440197016 (12h6m1.029542003s) [upload=12571]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:14.557737130 (12h8m14.147082107s) [upload=12571]: Done uploading segments... Aug 7 21:48:14.557758951 (12h8m14.147103938s) [upload=12571]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.146730545s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXXXXX..XX..X..............] duration: 5.841419767s FAIL (2025-08-07 21:58:19 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.841461608s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......X...............X..........X.................X.......] duration: 1.844829112s FAIL (2025-08-07 21:58:16 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 1.844872696s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 33 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X...X...................] duration: 1.951201222s FAIL (2025-08-07 21:58:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== line exceeded Aug 7 21:37:22.572699237 (11h57m22.162044224s) [upload=12432]: Starting upload Aug 7 21:37:22.572782937 (11h57m22.162127924s) [upload=12432]: Done starting upload Aug 7 21:37:22.572842260 (11h57m22.162187247s) [upload=12432]: Uploading segments... Aug 7 21:37:22.572885313 (11h57m22.1622303s) [upload=12432]: Got next segment. Inline: true Aug 7 21:37:22.572889591 (11h57m22.162234578s) [upload=12432]: Waiting for error group managing segments... Aug 7 21:37:22.572930138 (11h57m22.162275125s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:22.572934296 (11h57m22.162279283s) [upload=12432]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:22.572937121 (11h57m22.162282118s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.933954276 (11h57m51.523299253s) [upload=12432]: Done uploading segments... Aug 7 21:37:51.933970927 (11h57m51.523315914s) [upload=12432]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:28.745940745 (11h58m28.335285732s) [upload=12448]: Starting upload Aug 7 21:38:28.745997504 (11h58m28.335342491s) [upload=12448]: Done starting upload Aug 7 21:38:28.746040176 (11h58m28.335385163s) [upload=12448]: Uploading segments... Aug 7 21:38:28.746078458 (11h58m28.335423446s) [upload=12448]: Got next segment. Inline: true Aug 7 21:38:28.746081766 (11h58m28.335426743s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:38:28.746111923 (11h58m28.33545691s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:28.746115540 (11h58m28.335460527s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:28.746118205 (11h58m28.335463192s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:58.105315693 (11h58m57.694660691s) [upload=12448]: Done uploading segments... Aug 7 21:38:58.105342465 (11h58m57.694687452s) [upload=12448]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:27.137238449 (11h59m26.726583436s) [upload=12462]: Starting upload Aug 7 21:39:27.137294907 (11h59m26.726639894s) [upload=12462]: Done starting upload Aug 7 21:39:27.137337368 (11h59m26.726682355s) [upload=12462]: Uploading segments... Aug 7 21:39:27.137366043 (11h59m26.72671103s) [upload=12462]: Got next segment. Inline: true Aug 7 21:39:27.137370061 (11h59m26.726715048s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:39:27.137401901 (11h59m26.726746879s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:27.137405518 (11h59m26.726750495s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:27.137407813 (11h59m26.72675279s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.495186447 (11h59m56.084531434s) [upload=12462]: Done uploading segments... Aug 7 21:39:56.495206415 (11h59m56.084551402s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:01.914615953 (12h0m1.50396094s) [upload=12475]: Starting upload Aug 7 21:40:01.914684295 (12h0m1.504029282s) [upload=12475]: Done starting upload Aug 7 21:40:01.914741163 (12h0m1.50408616s) [upload=12475]: Uploading segments... Aug 7 21:40:01.914783103 (12h0m1.50412809s) [upload=12475]: Got next segment. Inline: true Aug 7 21:40:01.914788143 (12h0m1.504133131s) [upload=12475]: Waiting for error group managing segments... Aug 7 21:40:01.914833950 (12h0m1.504178937s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:01.914840073 (12h0m1.50418507s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:01.914846014 (12h0m1.504191011s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:31.268196986 (12h0m30.857542023s) [upload=12475]: Done uploading segments... Aug 7 21:40:31.268223536 (12h0m30.857568533s) [upload=12475]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:46.248438399 (12h0m45.837783386s) [upload=12487]: Starting upload Aug 7 21:40:46.248491410 (12h0m45.837836387s) [upload=12487]: Done starting upload Aug 7 21:40:46.248540414 (12h0m45.837885401s) [upload=12487]: Uploading segments... Aug 7 21:40:46.248572345 (12h0m45.837917332s) [upload=12487]: Got next segment. Inline: true Aug 7 21:40:46.248576133 (12h0m45.83792112s) [upload=12487]: Waiting for error group managing segments... Aug 7 21:40:46.248613013 (12h0m45.837958s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:46.248616850 (12h0m45.837961837s) [upload=12487]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:46.248619255 (12h0m45.837964242s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:55.261300379 (12h0m54.850645366s) [upload=12487]: Done uploading segments... Aug 7 21:40:55.261318514 (12h0m54.850663501s) [upload=12487]: Upload finished. err: <nil> Aug 7 21:42:00.373248983 (12h1m59.962594s) [upload=12509]: Starting upload Aug 7 21:42:00.373363813 (12h1m59.96270884s) [upload=12509]: Done starting upload Aug 7 21:42:00.373438045 (12h1m59.962783062s) [upload=12509]: Uploading segments... Aug 7 21:42:00.373501075 (12h1m59.962846083s) [upload=12509]: Got next segment. Inline: true Aug 7 21:42:00.373507809 (12h1m59.962852816s) [upload=12509]: Waiting for error group managing segments... Aug 7 21:42:00.373577642 (12h1m59.962922669s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:00.373584325 (12h1m59.962929342s) [upload=12509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:00.373590016 (12h1m59.962935053s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:29.724044552 (12h2m29.313389539s) [upload=12509]: Done uploading segments... Aug 7 21:42:29.724072415 (12h2m29.313417392s) [upload=12509]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:30.611135457 (12h2m30.200480444s) [upload=12516]: Starting upload Aug 7 21:42:30.611201894 (12h2m30.200546881s) [upload=12516]: Done starting upload Aug 7 21:42:30.611252732 (12h2m30.200597719s) [upload=12516]: Uploading segments... Aug 7 21:42:30.611338045 (12h2m30.200683032s) [upload=12516]: Got next segment. Inline: true Aug 7 21:42:30.611344076 (12h2m30.200689063s) [upload=12516]: Waiting for error group managing segments... Aug 7 21:42:30.611400083 (12h2m30.20074507s) [upload=12516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:30.611404922 (12h2m30.200749909s) [upload=12516]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:30.611408178 (12h2m30.200753166s) [upload=12516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:46.271341803 (12h2m45.860686781s) [upload=12516]: Done uploading segments... Aug 7 21:42:46.271358525 (12h2m45.860703512s) [upload=12516]: Upload finished. err: <nil> Aug 7 21:43:05.160958738 (12h3m4.750303725s) [upload=12528]: Starting upload Aug 7 21:43:05.161064639 (12h3m4.750409626s) [upload=12528]: Done starting upload Aug 7 21:43:05.161177936 (12h3m4.750522953s) [upload=12528]: Uploading segments... Aug 7 21:43:05.161261357 (12h3m4.750606344s) [upload=12528]: Got next segment. Inline: true Aug 7 21:43:05.161266376 (12h3m4.750611353s) [upload=12528]: Waiting for error group managing segments... Aug 7 21:43:05.161306081 (12h3m4.750651069s) [upload=12528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:05.161310580 (12h3m4.750655567s) [upload=12528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:05.161312864 (12h3m4.750657841s) [upload=12528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:17.899724589 (12h3m17.489069576s) [upload=12528]: Done uploading segments... Aug 7 21:43:17.899747483 (12h3m17.48909251s) [upload=12528]: Upload finished. err: <nil> Aug 7 21:43:34.894582431 (12h3m34.483927418s) [upload=12537]: Starting upload Aug 7 21:43:34.894647576 (12h3m34.483992564s) [upload=12537]: Done starting upload Aug 7 21:43:34.894699496 (12h3m34.484044483s) [upload=12537]: Uploading segments... Aug 7 21:43:34.894732940 (12h3m34.484077927s) [upload=12537]: Got next segment. Inline: true Aug 7 21:43:34.894736967 (12h3m34.484081954s) [upload=12537]: Waiting for error group managing segments... Aug 7 21:43:34.894770662 (12h3m34.484115649s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:34.894775050 (12h3m34.484120027s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:34.894777795 (12h3m34.484122782s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:35.191263901 (12h3m34.780608888s) [upload=12537]: Done uploading segments... Aug 7 21:43:35.191280001 (12h3m34.780624978s) [upload=12537]: Upload finished. err: <nil> Aug 7 21:44:27.756622626 (12h4m27.345967613s) [upload=12553]: Starting upload Aug 7 21:44:27.756679294 (12h4m27.346024281s) [upload=12553]: Done starting upload Aug 7 21:44:27.756721205 (12h4m27.346066192s) [upload=12553]: Uploading segments... Aug 7 21:44:27.756751423 (12h4m27.34609641s) [upload=12553]: Got next segment. Inline: true Aug 7 21:44:27.756754779 (12h4m27.346099766s) [upload=12553]: Waiting for error group managing segments... Aug 7 21:44:27.756785678 (12h4m27.346130655s) [upload=12553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:27.756789615 (12h4m27.346134602s) [upload=12553]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:27.756791790 (12h4m27.346136777s) [upload=12553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:57.108386877 (12h4m56.697731864s) [upload=12553]: Done uploading segments... Aug 7 21:44:57.108405372 (12h4m56.697750359s) [upload=12553]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.951220368s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......X..............] duration: 6.058705112s FAIL (2025-08-07 21:58:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.058729912s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXXX...X..X...............] duration: 5.963096538s FAIL (2025-08-07 21:58:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.963146716s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 35 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X........................] duration: 2.31480983s FAIL (2025-08-07 21:58:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.31485207s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 26 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXXXXXXXX.....X...............] duration: 6.146275297s FAIL (2025-08-07 21:58:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== t segment. Inline: false Aug 7 21:44:25.061589945 (12h4m24.650934922s) [upload=12552]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:25.061594724 (12h4m24.650939711s) [upload=12552]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:44:50.602885060 (12h4m50.192230047s) [seg_pos={0 0}, upload=12552]: Begin upload segment... Aug 7 21:44:50.602935446 (12h4m50.192280423s) [seg_pos={0 0}, upload=12552]: Done begin upload segment. Aug 7 21:44:50.603112565 (12h4m50.192457552s) [noise=true , address=79.127.205.233:4624 , piece_id=WEAYHGF7VE6W5ML3QAE3AITMEJGQFH4JQZ7RQKR7O4NKTPR6RZRA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.603695610 (12h4m50.193040597s) [noise=true , address=104.36.82.22:4529 , piece_id=ZTTBM5H6MHVMQ3OFVTVISN6WJ7WB3XBWI7YZKUXCFNQ6NO3D4CGA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.604300768 (12h4m50.193645755s) [noise=true , address=45.91.116.6:4002 , piece_id=AE5EL7NWD6FEQ7NXX2JRHB4MVZQDXRGQU3EER2X7COISFAX6P3XQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.604588519 (12h4m50.193933496s) [noise=true , address=104.36.82.56:4249 , piece_id=RD4B7IDIJV25L7EJNSYCZGW5QKJEM435E2N3UFUD44Q4S3DCBOXQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605153860 (12h4m50.194498837s) [noise=true , address=77.90.31.11:4562 , piece_id=XC6F2WLSXKQB7E7MBRNZBJKUSUFL6XNA5TVX5M3ZJI3ZZMK7UBLA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605455377 (12h4m50.194800364s) [noise=true , address=104.36.82.23:4466 , piece_id=5J5X2LSP6GCX2PNRIDZYFJPISWSX3YSRID5M2WU6Y5Z6LWRHLC4Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605743810 (12h4m50.195088797s) [noise=true , address=104.36.82.62:4420 , piece_id=UOO36PYVE5WFRRZLGYLJ6I6SV4EO7W2ZH2TIXJ2OIZZMY3LTHR2A, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.606547457 (12h4m50.195892444s) [noise=true , address=104.36.82.5:4842 , piece_id=E5DSJXN63RFX7EQ4ZTBMUESFW77EB3HQNA5KCQGRVRAA5XO2GN3Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.606982099 (12h4m50.196327086s) [noise=true , address=104.36.82.59:4301 , piece_id=VVR5CQWCPNPZUFEFCPB6ZL2KASWS5XO65FVDI474RKRZSJFDUQTA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.607295849 (12h4m50.196640836s) [noise=true , address=45.91.116.14:4198 , piece_id=2HRU3JZDQSNGD5NJZDLZSRG2WIO2Y2QBSOAY6KUEGUKDBWM53RSA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.607695593 (12h4m50.19704058s) [noise=true , address=104.36.82.6:4881 , piece_id=VUC33D7TLVFICDLA3334DEKBJE5E5P4HQTJU3VDQNMVYWVESSJWA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.608364103 (12h4m50.19770909s) [noise=true , address=104.36.82.2:4757 , piece_id=H6KNYP5HASPWQIHADTMEZDC5AE5HITCAY73XGNWVEXJOPDEZ3SHA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.608764288 (12h4m50.198109265s) [noise=true , address=104.36.82.46:4493 , piece_id=PS4S4OZ6HSITEJFPJDIBWQ5RGWTDRKMX3NCEWXCLC2PSP5Q2KPZQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609065284 (12h4m50.198410271s) [noise=true , address=104.36.82.47:4518 , piece_id=KJ3PJCT3EDH3WZBT7AWWKI6ABGVVAMKZTVJY7Q76FULC72TUWA6Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609312777 (12h4m50.198657755s) [noise=true , address=104.36.82.60:4349 , piece_id=GTTF32YMJWBY2GZEFZPIBZXZWNI7MBLUYCTQ3SRFG735PLWXTXBQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609589888 (12h4m50.198934875s) [noise=true , address=104.36.82.52:4116 , piece_id=7QA2Q5ACN43TQUOKWA34EZFSSQBE3OWNJTYODHHQ5GCVOA4QZOJA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.610295969 (12h4m50.199641056s) [noise=true , address=104.36.82.37:4678 , piece_id=VJO5JO3SJEXBQZC6PJ34DRKWEQKOPI33ZF4IHNNOOTSD7IBRR4PQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.610811305 (12h4m50.200156302s) [noise=true , address=64.49.13.169:4714 , piece_id=A6O3H7QLE6KQTHWUKOEROVHPEHVLG22KNF2KCV3HK2KCLXP7SYVA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:51.715421583 (12h4m51.30476657s) [upload=12552]: Next returned nil segment Aug 7 21:44:51.715438125 (12h4m51.304783112s) [upload=12552]: Waiting for error group managing segments... Aug 7 21:44:51.998131281 (12h4m51.587476268s) [noise=true , address=64.49.13.169:4714 , piece_id=A6O3H7QLE6KQTHWUKOEROVHPEHVLG22KNF2KCV3HK2KCLXP7SYVA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:51.999236335 (12h4m51.588581312s) [noise=true , address=79.127.205.233:4624 , piece_id=WEAYHGF7VE6W5ML3QAE3AITMEJGQFH4JQZ7RQKR7O4NKTPR6RZRA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.638299991 (12h4m52.227644978s) [noise=true , address=45.91.116.6:4002 , piece_id=AE5EL7NWD6FEQ7NXX2JRHB4MVZQDXRGQU3EER2X7COISFAX6P3XQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.639927565 (12h4m52.229272542s) [noise=true , address=104.36.82.6:4881 , piece_id=VUC33D7TLVFICDLA3334DEKBJE5E5P4HQTJU3VDQNMVYWVESSJWA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.643220585 (12h4m52.232565572s) [noise=true , address=104.36.82.2:4757 , piece_id=H6KNYP5HASPWQIHADTMEZDC5AE5HITCAY73XGNWVEXJOPDEZ3SHA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.711872926 (12h4m52.301217913s) [noise=true , address=104.36.82.56:4249 , piece_id=RD4B7IDIJV25L7EJNSYCZGW5QKJEM435E2N3UFUD44Q4S3DCBOXQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.763601896 (12h4m52.352946883s) [noise=true , address=104.36.82.22:4529 , piece_id=ZTTBM5H6MHVMQ3OFVTVISN6WJ7WB3XBWI7YZKUXCFNQ6NO3D4CGA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.768238797 (12h4m52.357583785s) [noise=true , address=104.36.82.60:4349 , piece_id=GTTF32YMJWBY2GZEFZPIBZXZWNI7MBLUYCTQ3SRFG735PLWXTXBQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.798543408 (12h4m52.387888395s) [noise=true , address=45.91.116.14:4198 , piece_id=2HRU3JZDQSNGD5NJZDLZSRG2WIO2Y2QBSOAY6KUEGUKDBWM53RSA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.861192704 (12h4m52.450537691s) [noise=true , address=77.90.31.11:4562 , piece_id=XC6F2WLSXKQB7E7MBRNZBJKUSUFL6XNA5TVX5M3ZJI3ZZMK7UBLA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:53.070218164 (12h4m52.659563151s) [noise=true , address=104.36.82.59:4301 , piece_id=VVR5CQWCPNPZUFEFCPB6ZL2KASWS5XO65FVDI474RKRZSJFDUQTA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:53.558930346 (12h4m53.148275333s) [noise=true , address=104.36.82.46:4493 , piece_id=PS4S4OZ6HSITEJFPJDIBWQ5RGWTDRKMX3NCEWXCLC2PSP5Q2KPZQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.132342777 (12h4m54.721687765s) [noise=true , address=104.36.82.37:4678 , piece_id=VJO5JO3SJEXBQZC6PJ34DRKWEQKOPI33ZF4IHNNOOTSD7IBRR4PQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.265236558 (12h4m54.854581545s) [noise=true , address=104.36.82.52:4116 , piece_id=7QA2Q5ACN43TQUOKWA34EZFSSQBE3OWNJTYODHHQ5GCVOA4QZOJA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.265254161 (12h4m54.854599138s) [seg_pos={0 0}, upload=12552]: Segment reached optimal threshold of 14 pieces. Aug 7 21:44:55.265482308 (12h4m54.854827295s) [noise=true , address=104.36.82.5:4842 , piece_id=E5DSJXN63RFX7EQ4ZTBMUESFW77EB3HQNA5KCQGRVRAA5XO2GN3Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn): protocol: expected piece hash; write tcp 188.166.225.170:57020->104.36.82.5:4842: use of closed network connection; write tcp 188.166.225.170:57020->104.36.82.5:4842: use of closed network connection; EOF Aug 7 21:44:55.265551430 (12h4m54.854896417s) [noise=true , address=104.36.82.47:4518 , piece_id=KJ3PJCT3EDH3WZBT7AWWKI6ABGVVAMKZTVJY7Q76FULC72TUWA6Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: expected piece hash; context canceled; EOF Aug 7 21:44:55.265766391 (12h4m54.855111378s) [noise=true , address=104.36.82.23:4466 , piece_id=5J5X2LSP6GCX2PNRIDZYFJPISWSX3YSRID5M2WU6Y5Z6LWRHLC4Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rkNnk9RUhKBJ3EsakqMRb3wYwrhRLhB4ZYbrdfKPDk8erRJpP): protocol: write tcp 188.166.225.170:49724->104.36.82.23:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:44:55.265824172 (12h4m54.855169159s) [noise=true , address=104.36.82.62:4420 , piece_id=UOO36PYVE5WFRRZLGYLJ6I6SV4EO7W2ZH2TIXJ2OIZZMY3LTHR2A, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SWVQwL5PxDLaSXADwTrAYS2hVcNBuYAA6ReWuRM3UZa57QyYi): protocol: read tcp 188.166.225.170:33092->104.36.82.62:4420: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:44:55.265833430 (12h4m54.855178417s) [seg_pos={0 0}, upload=12552]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:44:55.265888796 (12h4m54.855233783s) [upload=12552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:44:55.265892303 (12h4m54.85523728s) [upload=12552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:46.088275738 (12h8m45.677620725s) [upload=12552]: Done uploading segments... Aug 7 21:48:46.088301307 (12h8m45.677646284s) [upload=12552]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.146301507s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 33 out of 60 is failed (threshold=3) [....X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...X..................] duration: 1.866086204s FAIL (2025-08-07 21:58:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.86611005s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 29 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXXXXXXXXX..XX..X.............] duration: 5.655882317s FAIL (2025-08-07 21:57:55 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.655905392s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.......................] duration: 2.414286523s FAIL (2025-08-07 21:57:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.414308225s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [.........................XX.XX..X.XX........................] duration: 231.394418ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:57:44 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 231.416432ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X......X..X.............X..X....................X....] duration: 7.155427691s FAIL (2025-08-07 21:57:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 57 (12h15m32.758009944s) [noise=true , address=104.36.82.54:4166 , piece_id=5SBUSGFVYPNFGILW6OTNLGJQG7CNI7GYYEJQJMITPKFZ2MAK5QRQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.168945643 (12h15m32.75829063s) [noise=true , address=104.36.82.12:4179 , piece_id=KFVQX32X6I7QBU552L4NSPZG3JTVT24YESONPWGM6UCTR4ZWBUKQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.168966093 (12h15m32.75831109s) [noise=true , address=104.36.82.43:4850 , piece_id=JH2ASD4NQAQMOBOZMDGFKVPPOS2P26QIFSYDNHYZN24RP52E4XLA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169201393 (12h15m32.75854638s) [noise=true , address=104.36.82.36:4633 , piece_id=6RL3INCLYJCEH53JOFJDL2MMEXGQ2G4T33PKZPJ3W2FG5QMCFLCQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169453466 (12h15m32.758798453s) [noise=true , address=104.36.82.47:4520 , piece_id=K75KQHBYH7QTZDCFM7ZW55EUO2W3UN2NEIKUMNCMQWHYYNOWLL3A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169588975 (12h15m32.758933962s) [noise=true , address=104.36.82.9:4014 , piece_id=D3NLTEDBRB4M7GIMPYCDNQ66HYLAZGSSBRM7SJWU45LPTCRMYMCQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170078472 (12h15m32.759423459s) [noise=true , address=104.36.82.39:4739 , piece_id=QGKQ3NJO7OTN5O2QQU43KYJXARJW6JQMSCONEJR35AEBBNRORJCA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170313160 (12h15m32.759658147s) [noise=true , address=104.36.82.62:4429 , piece_id=RCPKZHWWS56QSEVG6L25B3FMQOU5ZES3BNAEYIUYQ3PLQV3FDMVQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170515587 (12h15m32.759860574s) [noise=true , address=79.127.201.215:4353 , piece_id=II3JPNVAATS6APC4JGPPOJRHQRMHS5EYGLLEEAMTOPHPEGRQUF7A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170753382 (12h15m32.760098359s) [noise=true , address=45.91.116.8:4065 , piece_id=OEVAVKCHFYMJMGBKUKF5F34O3NH47DG45QJHHNWMOD7TDRMCQLAQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170965789 (12h15m32.760310776s) [noise=true , address=77.90.31.14:4678 , piece_id=7WZVWEAFKI4ZEIHTZOJNMAQWLBVPPLB7U4JNYZQNXCVRC2S4BUBA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171167525 (12h15m32.760512512s) [noise=true , address=104.36.82.45:4926 , piece_id=S3EH75AGZ3QVGR6RM3RI3SCHXZJPE7LORX2QQA5SCEADV3RDYM6A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171354372 (12h15m32.760699359s) [noise=true , address=64.49.13.154:4935 , piece_id=O6TZCBWBHVESLRX76EFTUT4U4OIKFFVXO5UKLMWAJA2BDTHIHN2A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171604611 (12h15m32.760949598s) [noise=true , address=104.36.82.44:4894 , piece_id=WCLKE2VTORDUBR5II5IQ5OTYMNHCMP2OYAIBHG3GV5P5U3FCO7HQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171788884 (12h15m32.761133861s) [noise=true , address=104.36.82.37:4674 , piece_id=AN3RCVXXXNEJUD33D4W47FDFYJQ7X5ZK7H4BJD6NR4PJOJSCVV6A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.181382449 (12h15m32.770727426s) [noise=true , address=104.36.82.23:4474 , piece_id=WM2SVCDZEFAN6VXQ2PWKBZGUCHIHNJ52RNKJKUNBMJ3AHFXUT3PA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:34.281100029 (12h15m33.870445006s) [upload=12734]: Next returned nil segment Aug 7 21:55:34.281113294 (12h15m33.870458281s) [upload=12734]: Waiting for error group managing segments... Aug 7 21:55:34.581166654 (12h15m34.170511631s) [noise=true , address=64.49.13.154:4935 , piece_id=O6TZCBWBHVESLRX76EFTUT4U4OIKFFVXO5UKLMWAJA2BDTHIHN2A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.075145374 (12h15m34.664490361s) [noise=true , address=79.127.201.215:4353 , piece_id=II3JPNVAATS6APC4JGPPOJRHQRMHS5EYGLLEEAMTOPHPEGRQUF7A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.117523204 (12h15m34.706868191s) [noise=true , address=77.90.31.14:4678 , piece_id=7WZVWEAFKI4ZEIHTZOJNMAQWLBVPPLB7U4JNYZQNXCVRC2S4BUBA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.189259324 (12h15m34.778604311s) [noise=true , address=104.36.82.9:4014 , piece_id=D3NLTEDBRB4M7GIMPYCDNQ66HYLAZGSSBRM7SJWU45LPTCRMYMCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.208589096 (12h15m34.797934083s) [noise=true , address=104.36.82.23:4474 , piece_id=WM2SVCDZEFAN6VXQ2PWKBZGUCHIHNJ52RNKJKUNBMJ3AHFXUT3PA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.240233681 (12h15m34.829578668s) [noise=true , address=104.36.82.58:4265 , piece_id=XMDZZK372B4DNLPJRKD2BM7OI7JNQSZ6ABLSXNPZEH2WT73AT7TA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.249509658 (12h15m34.838854645s) [noise=true , address=104.36.82.39:4739 , piece_id=QGKQ3NJO7OTN5O2QQU43KYJXARJW6JQMSCONEJR35AEBBNRORJCA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.254602933 (12h15m34.84394792s) [noise=true , address=104.36.82.12:4179 , piece_id=KFVQX32X6I7QBU552L4NSPZG3JTVT24YESONPWGM6UCTR4ZWBUKQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.255644015 (12h15m34.844989002s) [noise=true , address=104.36.82.59:4322 , piece_id=O6L7GOHWQOA6VAMNJATA232B7BEQS66XKSZXUAQLX3KYNQXTMUCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.260038844 (12h15m34.849383831s) [noise=true , address=45.91.116.8:4065 , piece_id=OEVAVKCHFYMJMGBKUKF5F34O3NH47DG45QJHHNWMOD7TDRMCQLAQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.160031235 (12h15m35.749376222s) [noise=true , address=104.36.82.43:4850 , piece_id=JH2ASD4NQAQMOBOZMDGFKVPPOS2P26QIFSYDNHYZN24RP52E4XLA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.204255979 (12h15m35.793600956s) [noise=true , address=104.36.82.62:4429 , piece_id=RCPKZHWWS56QSEVG6L25B3FMQOU5ZES3BNAEYIUYQ3PLQV3FDMVQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.608134744 (12h15m36.197479731s) [noise=true , address=104.36.82.37:4674 , piece_id=AN3RCVXXXNEJUD33D4W47FDFYJQ7X5ZK7H4BJD6NR4PJOJSCVV6A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.985735083 (12h15m36.57508007s) [noise=true , address=104.36.82.45:4926 , piece_id=S3EH75AGZ3QVGR6RM3RI3SCHXZJPE7LORX2QQA5SCEADV3RDYM6A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.985758668 (12h15m36.575103655s) [seg_pos={0 0}, upload=12734]: Segment reached optimal threshold of 14 pieces. Aug 7 21:55:36.985891332 (12h15m36.575236319s) [noise=true , address=104.36.82.36:4633 , piece_id=6RL3INCLYJCEH53JOFJDL2MMEXGQ2G4T33PKZPJ3W2FG5QMCFLCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dvBPvb9YEtM4NieVCXL71Pak5zwiMg14MSXH4RuLZts4dByG8): protocol: expected piece hash; context canceled; EOF Aug 7 21:55:36.986091084 (12h15m36.575436071s) [noise=true , address=104.36.82.47:4520 , piece_id=K75KQHBYH7QTZDCFM7ZW55EUO2W3UN2NEIKUMNCMQWHYYNOWLL3A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy): protocol: write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection; write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection Aug 7 21:55:36.986139326 (12h15m36.575484333s) [noise=true , address=104.36.82.54:4166 , piece_id=5SBUSGFVYPNFGILW6OTNLGJQG7CNI7GYYEJQJMITPKFZ2MAK5QRQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv): protocol: expected piece hash; context canceled; EOF Aug 7 21:55:36.986268794 (12h15m36.575613791s) [noise=true , address=104.36.82.44:4894 , piece_id=WCLKE2VTORDUBR5II5IQ5OTYMNHCMP2OYAIBHG3GV5P5U3FCO7HQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44): protocol: read tcp 188.166.225.170:34542->104.36.82.44:4894: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:55:36.986293281 (12h15m36.575638268s) [seg_pos={0 0}, upload=12734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:55:36.986385397 (12h15m36.575730394s) [upload=12734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:55:36.986391038 (12h15m36.575736025s) [upload=12734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:55:37.437633211 (12h15m37.026978238s) [upload=12734]: Done uploading segments... Aug 7 21:55:37.437656937 (12h15m37.027001924s) [upload=12734]: Upload finished. err: <nil> Aug 7 21:55:48.800457325 (12h15m48.389802332s) [upload=12741]: Starting upload Aug 7 21:55:48.800589529 (12h15m48.389934526s) [upload=12741]: Done starting upload Aug 7 21:55:48.800688879 (12h15m48.390033866s) [upload=12741]: Uploading segments... Aug 7 21:55:48.801167825 (12h15m48.390512802s) [upload=12741]: Got next segment. Inline: false Aug 7 21:55:48.801200477 (12h15m48.390545464s) [upload=12741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:55:48.801204775 (12h15m48.390549762s) [upload=12741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:55:49.015107648 (12h15m48.604452635s) [upload=12741]: Next segment err: metaclient: Too Many Requests Aug 7 21:55:49.015124520 (12h15m48.604469507s) [upload=12741]: Done uploading segments... Aug 7 21:55:49.015136393 (12h15m48.60448138s) [upload=12741]: Upload finished. err: metaclient: Too Many Requests Duration: 7.155452629s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)