Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 6.631678641s FAIL (2025-08-17 04:31:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.631783434s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................XXXX.........] duration: 5.989492379s FAIL (2025-08-17 04:31:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.98951838s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................XXXX..........] duration: 6.255147236s FAIL (2025-08-17 04:31:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.255176792s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [............X.X......................X..........XXXX.X.X....] duration: 5.628140196s FAIL (2025-08-17 04:30:59 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 17 04:27:30.016543050 (203h1m24.447623181s) [upload=211178]: Starting upload Aug 17 04:27:30.016583197 (203h1m24.447663328s) [upload=211178]: Done starting upload Aug 17 04:27:30.016768071 (203h1m24.447848202s) [upload=211178]: Uploading segments... Aug 17 04:27:30.016798940 (203h1m24.447879071s) [upload=211178]: Got next segment. Inline: false Aug 17 04:27:30.016828156 (203h1m24.447908277s) [upload=211178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:27:30.016832424 (203h1m24.447912555s) [upload=211178]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:27:30.463480034 (203h1m24.894560165s) [seg_pos={0 0}, upload=211178]: Begin upload segment... Aug 17 04:27:30.463526713 (203h1m24.894606844s) [seg_pos={0 0}, upload=211178]: Done begin upload segment. Aug 17 04:27:30.463547864 (203h1m24.894627995s) [noise=true , address=77.90.31.12:4610 , piece_id=5HQU72EXU3FU6ZQNTDQDEXWYPATMLKYKEEMGSKNTPAI4JCZAJ5IQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.463559155 (203h1m24.894639286s) [noise=true , address=104.36.82.8:4992 , piece_id=PA7YXTWXXC5SL3UUEWAF7BAJCHDOKZ3BDGZIASS62342EOWUH7IQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.463911099 (203h1m24.89499122s) [noise=true , address=77.90.31.15:4711 , piece_id=CVYW4ML4GFE5IC5SV4G3VEKI47OM5L5O4D7VWKPTUATR2MDIDHBQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.463926858 (203h1m24.895006989s) [noise=true , address=104.36.82.22:4514 , piece_id=CI5POH6B53BJVI6ON4SR3OZSIH33SUFDRA5CJL25CLHLUDZWAS2A, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.464210091 (203h1m24.895290222s) [noise=true , address=77.90.31.14:4678 , piece_id=O3WNM7XRGUZF2WE64X5CCXX5CASWNXVX453NN2BQVDYPZURIOWXA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.464478404 (203h1m24.895558536s) [noise=true , address=104.36.82.60:4369 , piece_id=SZVCNKEW3RAMIAV3XKLIPHYG5YCSTOV2P76NCMS2A2VBMP6R6DBQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.464940458 (203h1m24.896020589s) [noise=true , address=79.127.201.215:4358 , piece_id=RIAIZRP4SCLM2DCKL2QDNHDQEZXURQS5SKMVCUUMHDBXLO7DGJTA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465180658 (203h1m24.896260779s) [noise=true , address=77.90.31.11:4578 , piece_id=PFYRUL44F7FGFRCMUNKVVK4AFP4CSX7Y55Y6O54UCRKNUHWZ7OIQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465394196 (203h1m24.896474327s) [noise=true , address=77.90.31.21:4899 , piece_id=JD3QWQLDB55IJIQ7WUWC3GFTQHRF4UBFZMBGQO4BC7DIMIVSM3KA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465601532 (203h1m24.896681664s) [noise=true , address=104.36.82.2:4762 , piece_id=5MFQVM2XDPNGK4JVFUS7GXCA3M5FDPMG4VHVVXDQ6KLL3F5VSEBA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465820732 (203h1m24.896900853s) [noise=true , address=104.36.82.23:4466 , piece_id=TTBPYPOV6G6VPYTH3BZBA5I2FVW4ORLGAE34JSE6IP4W3UOR6ERQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466052235 (203h1m24.897132366s) [noise=true , address=104.36.82.34:4549 , piece_id=EUAUXMTSRI4UNN4T6XBFL5GXSAP4W4WJ2Z6YBEOGLWLXBKGDYF4Q, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466300020 (203h1m24.897380151s) [noise=true , address=104.36.82.3:4776 , piece_id=ZGFMY6UVQIGXG4XZBWLKMJRWGHMV4ONJ7BBTBXUQOOSTZL7MGK3Q, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466559126 (203h1m24.897639257s) [noise=true , address=104.36.82.5:4872 , piece_id=QDPEEVMTYH35DAERQVGB55IBSNG7NEZCMFNN2GS5JPLGDC4L54IQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466765811 (203h1m24.897845942s) [noise=true , address=79.127.205.233:4646 , piece_id=PQVSNUKLSQ3Q7VLDAGOST7ZIDT2BHVBS43IUZ4X3V7NVFYE67UNA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466965002 (203h1m24.898045123s) [noise=true , address=77.90.31.19:4860 , piece_id=54C73ZPECTG4GPKYMTVOJVOIRRLF6XNE3F36FK632WODFNRBJVXA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.467163783 (203h1m24.898243914s) [noise=true , address=45.91.116.7:4018 , piece_id=GEKZFUTPVL6CJHTRK25E5EL3KF4Y2LASY6CLKA2G5YNFAWFGXBQQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.467361531 (203h1m24.898441662s) [noise=true , address=104.36.82.38:4714 , piece_id=V247PLY3PWQLOGIV6EPXFJDTEI6A2WKSV6HUDNDB4N7ZQSJ7HFDA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:31.608869595 (203h1m26.039949716s) [upload=211178]: Next returned nil segment Aug 17 04:27:31.608891085 (203h1m26.039971266s) [upload=211178]: Waiting for error group managing segments... Aug 17 04:27:31.900075977 (203h1m26.331156108s) [noise=true , address=79.127.205.233:4646 , piece_id=PQVSNUKLSQ3Q7VLDAGOST7ZIDT2BHVBS43IUZ4X3V7NVFYE67UNA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.352306149 (203h1m26.78338628s) [noise=true , address=104.36.82.60:4369 , piece_id=SZVCNKEW3RAMIAV3XKLIPHYG5YCSTOV2P76NCMS2A2VBMP6R6DBQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.357051599 (203h1m26.78813172s) [noise=true , address=104.36.82.8:4992 , piece_id=PA7YXTWXXC5SL3UUEWAF7BAJCHDOKZ3BDGZIASS62342EOWUH7IQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.368906113 (203h1m26.799986244s) [noise=true , address=104.36.82.2:4762 , piece_id=5MFQVM2XDPNGK4JVFUS7GXCA3M5FDPMG4VHVVXDQ6KLL3F5VSEBA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.399599454 (203h1m26.830679585s) [noise=true , address=104.36.82.5:4872 , piece_id=QDPEEVMTYH35DAERQVGB55IBSNG7NEZCMFNN2GS5JPLGDC4L54IQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.422356189 (203h1m26.85343632s) [noise=true , address=104.36.82.38:4714 , piece_id=V247PLY3PWQLOGIV6EPXFJDTEI6A2WKSV6HUDNDB4N7ZQSJ7HFDA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.423908750 (203h1m26.854988881s) [noise=true , address=77.90.31.11:4578 , piece_id=PFYRUL44F7FGFRCMUNKVVK4AFP4CSX7Y55Y6O54UCRKNUHWZ7OIQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.440077859 (203h1m26.87115801s) [noise=true , address=104.36.82.22:4514 , piece_id=CI5POH6B53BJVI6ON4SR3OZSIH33SUFDRA5CJL25CLHLUDZWAS2A, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.440882980 (203h1m26.871963111s) [noise=true , address=104.36.82.34:4549 , piece_id=EUAUXMTSRI4UNN4T6XBFL5GXSAP4W4WJ2Z6YBEOGLWLXBKGDYF4Q, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.454743903 (203h1m26.885824034s) [noise=true , address=79.127.201.215:4358 , piece_id=RIAIZRP4SCLM2DCKL2QDNHDQEZXURQS5SKMVCUUMHDBXLO7DGJTA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.464618066 (203h1m26.895698198s) [noise=true , address=77.90.31.14:4678 , piece_id=O3WNM7XRGUZF2WE64X5CCXX5CASWNXVX453NN2BQVDYPZURIOWXA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.468942671 (203h1m26.900022793s) [noise=true , address=45.91.116.7:4018 , piece_id=GEKZFUTPVL6CJHTRK25E5EL3KF4Y2LASY6CLKA2G5YNFAWFGXBQQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.495852825 (203h1m26.926932956s) [noise=true , address=104.36.82.3:4776 , piece_id=ZGFMY6UVQIGXG4XZBWLKMJRWGHMV4ONJ7BBTBXUQOOSTZL7MGK3Q, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.504721685 (203h1m26.935801816s) [noise=true , address=104.36.82.23:4466 , piece_id=TTBPYPOV6G6VPYTH3BZBA5I2FVW4ORLGAE34JSE6IP4W3UOR6ERQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.504743026 (203h1m26.935823147s) [seg_pos={0 0}, upload=211178]: Segment reached optimal threshold of 14 pieces. Aug 17 04:27:32.504932047 (203h1m26.936012178s) [noise=true , address=77.90.31.15:4711 , piece_id=CVYW4ML4GFE5IC5SV4G3VEKI47OM5L5O4D7VWKPTUATR2MDIDHBQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Aug 17 04:27:32.505005558 (203h1m26.936085689s) [noise=true , address=77.90.31.21:4899 , piece_id=JD3QWQLDB55IJIQ7WUWC3GFTQHRF4UBFZMBGQO4BC7DIMIVSM3KA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG): protocol: write tcp 188.166.225.170:55756->77.90.31.21:4899: use of closed network connection; write tcp 188.166.225.170:55756->77.90.31.21:4899: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55756->77.90.31.21:4899: use of closed network connection Aug 17 04:27:32.505020727 (203h1m26.936100869s) [noise=true , address=77.90.31.19:4860 , piece_id=54C73ZPECTG4GPKYMTVOJVOIRRLF6XNE3F36FK632WODFNRBJVXA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF Aug 17 04:27:32.505052127 (203h1m26.936132248s) [noise=true , address=77.90.31.12:4610 , piece_id=5HQU72EXU3FU6ZQNTDQDEXWYPATMLKYKEEMGSKNTPAI4JCZAJ5IQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: write tcp 188.166.225.170:40394->77.90.31.12:4610: use of closed network connection; write tcp 188.166.225.170:40394->77.90.31.12:4610: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40394->77.90.31.12:4610: use of closed network connection Aug 17 04:27:32.505062547 (203h1m26.936142678s) [seg_pos={0 0}, upload=211178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:27:32.505114917 (203h1m26.936195048s) [upload=211178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:27:32.505118574 (203h1m26.936198705s) [upload=211178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:27:32.796719122 (203h1m27.227799253s) [upload=211178]: Done uploading segments... Aug 17 04:27:32.796750893 (203h1m27.227831024s) [upload=211178]: Upload finished. err: <nil> Duration: 5.628246558s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................XXXX.........] duration: 6.47738045s FAIL (2025-08-17 04:30:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.477404145s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................XXX...........] duration: 5.620886716s FAIL (2025-08-17 04:30:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.620910811s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 6.472102024s FAIL (2025-08-17 04:30:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.472124327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X.........X.........................................] duration: 1.806926399s FAIL (2025-08-17 04:30:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 03:56:48.655583790 (202h30m43.086663921s) [upload=210657]: Starting upload Aug 17 03:56:48.655627825 (202h30m43.086707956s) [upload=210657]: Done starting upload Aug 17 03:56:48.655720953 (202h30m43.086801084s) [upload=210657]: Uploading segments... Aug 17 03:56:48.655806216 (202h30m43.086886347s) [upload=210657]: Got next segment. Inline: true Aug 17 03:56:48.655811276 (202h30m43.086891407s) [upload=210657]: Waiting for error group managing segments... Aug 17 03:56:48.655854448 (202h30m43.086934579s) [upload=210657]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 03:56:48.655859227 (202h30m43.086939358s) [upload=210657]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 03:56:48.655861832 (202h30m43.086941963s) [upload=210657]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 03:56:48.869012190 (202h30m43.300092331s) [upload=210657]: Done uploading segments... Aug 17 03:56:48.869031557 (202h30m43.300111708s) [upload=210657]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.806954304s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................XXX..........] duration: 5.765283127s FAIL (2025-08-17 04:30:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.765315259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............X.X......................X..........XXXX.X.X...] duration: 5.817559258s FAIL (2025-08-17 04:30:07 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 17 04:27:30.016543050 (203h1m24.447623181s) [upload=211178]: Starting upload Aug 17 04:27:30.016583197 (203h1m24.447663328s) [upload=211178]: Done starting upload Aug 17 04:27:30.016768071 (203h1m24.447848202s) [upload=211178]: Uploading segments... Aug 17 04:27:30.016798940 (203h1m24.447879071s) [upload=211178]: Got next segment. Inline: false Aug 17 04:27:30.016828156 (203h1m24.447908277s) [upload=211178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:27:30.016832424 (203h1m24.447912555s) [upload=211178]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:27:30.463480034 (203h1m24.894560165s) [seg_pos={0 0}, upload=211178]: Begin upload segment... Aug 17 04:27:30.463526713 (203h1m24.894606844s) [seg_pos={0 0}, upload=211178]: Done begin upload segment. Aug 17 04:27:30.463547864 (203h1m24.894627995s) [noise=true , address=77.90.31.12:4610 , piece_id=5HQU72EXU3FU6ZQNTDQDEXWYPATMLKYKEEMGSKNTPAI4JCZAJ5IQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.463559155 (203h1m24.894639286s) [noise=true , address=104.36.82.8:4992 , piece_id=PA7YXTWXXC5SL3UUEWAF7BAJCHDOKZ3BDGZIASS62342EOWUH7IQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.463911099 (203h1m24.89499122s) [noise=true , address=77.90.31.15:4711 , piece_id=CVYW4ML4GFE5IC5SV4G3VEKI47OM5L5O4D7VWKPTUATR2MDIDHBQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.463926858 (203h1m24.895006989s) [noise=true , address=104.36.82.22:4514 , piece_id=CI5POH6B53BJVI6ON4SR3OZSIH33SUFDRA5CJL25CLHLUDZWAS2A, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.464210091 (203h1m24.895290222s) [noise=true , address=77.90.31.14:4678 , piece_id=O3WNM7XRGUZF2WE64X5CCXX5CASWNXVX453NN2BQVDYPZURIOWXA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.464478404 (203h1m24.895558536s) [noise=true , address=104.36.82.60:4369 , piece_id=SZVCNKEW3RAMIAV3XKLIPHYG5YCSTOV2P76NCMS2A2VBMP6R6DBQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.464940458 (203h1m24.896020589s) [noise=true , address=79.127.201.215:4358 , piece_id=RIAIZRP4SCLM2DCKL2QDNHDQEZXURQS5SKMVCUUMHDBXLO7DGJTA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465180658 (203h1m24.896260779s) [noise=true , address=77.90.31.11:4578 , piece_id=PFYRUL44F7FGFRCMUNKVVK4AFP4CSX7Y55Y6O54UCRKNUHWZ7OIQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465394196 (203h1m24.896474327s) [noise=true , address=77.90.31.21:4899 , piece_id=JD3QWQLDB55IJIQ7WUWC3GFTQHRF4UBFZMBGQO4BC7DIMIVSM3KA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465601532 (203h1m24.896681664s) [noise=true , address=104.36.82.2:4762 , piece_id=5MFQVM2XDPNGK4JVFUS7GXCA3M5FDPMG4VHVVXDQ6KLL3F5VSEBA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465820732 (203h1m24.896900853s) [noise=true , address=104.36.82.23:4466 , piece_id=TTBPYPOV6G6VPYTH3BZBA5I2FVW4ORLGAE34JSE6IP4W3UOR6ERQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466052235 (203h1m24.897132366s) [noise=true , address=104.36.82.34:4549 , piece_id=EUAUXMTSRI4UNN4T6XBFL5GXSAP4W4WJ2Z6YBEOGLWLXBKGDYF4Q, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466300020 (203h1m24.897380151s) [noise=true , address=104.36.82.3:4776 , piece_id=ZGFMY6UVQIGXG4XZBWLKMJRWGHMV4ONJ7BBTBXUQOOSTZL7MGK3Q, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466559126 (203h1m24.897639257s) [noise=true , address=104.36.82.5:4872 , piece_id=QDPEEVMTYH35DAERQVGB55IBSNG7NEZCMFNN2GS5JPLGDC4L54IQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466765811 (203h1m24.897845942s) [noise=true , address=79.127.205.233:4646 , piece_id=PQVSNUKLSQ3Q7VLDAGOST7ZIDT2BHVBS43IUZ4X3V7NVFYE67UNA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466965002 (203h1m24.898045123s) [noise=true , address=77.90.31.19:4860 , piece_id=54C73ZPECTG4GPKYMTVOJVOIRRLF6XNE3F36FK632WODFNRBJVXA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.467163783 (203h1m24.898243914s) [noise=true , address=45.91.116.7:4018 , piece_id=GEKZFUTPVL6CJHTRK25E5EL3KF4Y2LASY6CLKA2G5YNFAWFGXBQQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.467361531 (203h1m24.898441662s) [noise=true , address=104.36.82.38:4714 , piece_id=V247PLY3PWQLOGIV6EPXFJDTEI6A2WKSV6HUDNDB4N7ZQSJ7HFDA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:31.608869595 (203h1m26.039949716s) [upload=211178]: Next returned nil segment Aug 17 04:27:31.608891085 (203h1m26.039971266s) [upload=211178]: Waiting for error group managing segments... Aug 17 04:27:31.900075977 (203h1m26.331156108s) [noise=true , address=79.127.205.233:4646 , piece_id=PQVSNUKLSQ3Q7VLDAGOST7ZIDT2BHVBS43IUZ4X3V7NVFYE67UNA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.352306149 (203h1m26.78338628s) [noise=true , address=104.36.82.60:4369 , piece_id=SZVCNKEW3RAMIAV3XKLIPHYG5YCSTOV2P76NCMS2A2VBMP6R6DBQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.357051599 (203h1m26.78813172s) [noise=true , address=104.36.82.8:4992 , piece_id=PA7YXTWXXC5SL3UUEWAF7BAJCHDOKZ3BDGZIASS62342EOWUH7IQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.368906113 (203h1m26.799986244s) [noise=true , address=104.36.82.2:4762 , piece_id=5MFQVM2XDPNGK4JVFUS7GXCA3M5FDPMG4VHVVXDQ6KLL3F5VSEBA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.399599454 (203h1m26.830679585s) [noise=true , address=104.36.82.5:4872 , piece_id=QDPEEVMTYH35DAERQVGB55IBSNG7NEZCMFNN2GS5JPLGDC4L54IQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.422356189 (203h1m26.85343632s) [noise=true , address=104.36.82.38:4714 , piece_id=V247PLY3PWQLOGIV6EPXFJDTEI6A2WKSV6HUDNDB4N7ZQSJ7HFDA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.423908750 (203h1m26.854988881s) [noise=true , address=77.90.31.11:4578 , piece_id=PFYRUL44F7FGFRCMUNKVVK4AFP4CSX7Y55Y6O54UCRKNUHWZ7OIQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.440077859 (203h1m26.87115801s) [noise=true , address=104.36.82.22:4514 , piece_id=CI5POH6B53BJVI6ON4SR3OZSIH33SUFDRA5CJL25CLHLUDZWAS2A, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.440882980 (203h1m26.871963111s) [noise=true , address=104.36.82.34:4549 , piece_id=EUAUXMTSRI4UNN4T6XBFL5GXSAP4W4WJ2Z6YBEOGLWLXBKGDYF4Q, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.454743903 (203h1m26.885824034s) [noise=true , address=79.127.201.215:4358 , piece_id=RIAIZRP4SCLM2DCKL2QDNHDQEZXURQS5SKMVCUUMHDBXLO7DGJTA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.464618066 (203h1m26.895698198s) [noise=true , address=77.90.31.14:4678 , piece_id=O3WNM7XRGUZF2WE64X5CCXX5CASWNXVX453NN2BQVDYPZURIOWXA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.468942671 (203h1m26.900022793s) [noise=true , address=45.91.116.7:4018 , piece_id=GEKZFUTPVL6CJHTRK25E5EL3KF4Y2LASY6CLKA2G5YNFAWFGXBQQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.495852825 (203h1m26.926932956s) [noise=true , address=104.36.82.3:4776 , piece_id=ZGFMY6UVQIGXG4XZBWLKMJRWGHMV4ONJ7BBTBXUQOOSTZL7MGK3Q, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.504721685 (203h1m26.935801816s) [noise=true , address=104.36.82.23:4466 , piece_id=TTBPYPOV6G6VPYTH3BZBA5I2FVW4ORLGAE34JSE6IP4W3UOR6ERQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.504743026 (203h1m26.935823147s) [seg_pos={0 0}, upload=211178]: Segment reached optimal threshold of 14 pieces. Aug 17 04:27:32.504932047 (203h1m26.936012178s) [noise=true , address=77.90.31.15:4711 , piece_id=CVYW4ML4GFE5IC5SV4G3VEKI47OM5L5O4D7VWKPTUATR2MDIDHBQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Aug 17 04:27:32.505005558 (203h1m26.936085689s) [noise=true , address=77.90.31.21:4899 , piece_id=JD3QWQLDB55IJIQ7WUWC3GFTQHRF4UBFZMBGQO4BC7DIMIVSM3KA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG): protocol: write tcp 188.166.225.170:55756->77.90.31.21:4899: use of closed network connection; write tcp 188.166.225.170:55756->77.90.31.21:4899: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55756->77.90.31.21:4899: use of closed network connection Aug 17 04:27:32.505020727 (203h1m26.936100869s) [noise=true , address=77.90.31.19:4860 , piece_id=54C73ZPECTG4GPKYMTVOJVOIRRLF6XNE3F36FK632WODFNRBJVXA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF Aug 17 04:27:32.505052127 (203h1m26.936132248s) [noise=true , address=77.90.31.12:4610 , piece_id=5HQU72EXU3FU6ZQNTDQDEXWYPATMLKYKEEMGSKNTPAI4JCZAJ5IQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: write tcp 188.166.225.170:40394->77.90.31.12:4610: use of closed network connection; write tcp 188.166.225.170:40394->77.90.31.12:4610: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40394->77.90.31.12:4610: use of closed network connection Aug 17 04:27:32.505062547 (203h1m26.936142678s) [seg_pos={0 0}, upload=211178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:27:32.505114917 (203h1m26.936195048s) [upload=211178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:27:32.505118574 (203h1m26.936198705s) [upload=211178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:27:32.796719122 (203h1m27.227799253s) [upload=211178]: Done uploading segments... Aug 17 04:27:32.796750893 (203h1m27.227831024s) [upload=211178]: Upload finished. err: <nil> Duration: 5.817588824s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................XXXX........] duration: 6.055100324s FAIL (2025-08-17 04:30:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.055130672s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 6.381024889s FAIL (2025-08-17 04:29:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.381053865s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................XXXX........] duration: 6.564576173s FAIL (2025-08-17 04:29:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.56460036s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................XXX.........] duration: 5.868282317s FAIL (2025-08-17 04:29:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.868307243s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [..............X.X......................X..........XXXX.X.X..] duration: 5.588159876s FAIL (2025-08-17 04:29:29 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 17 04:27:30.016543050 (203h1m24.447623181s) [upload=211178]: Starting upload Aug 17 04:27:30.016583197 (203h1m24.447663328s) [upload=211178]: Done starting upload Aug 17 04:27:30.016768071 (203h1m24.447848202s) [upload=211178]: Uploading segments... Aug 17 04:27:30.016798940 (203h1m24.447879071s) [upload=211178]: Got next segment. Inline: false Aug 17 04:27:30.016828156 (203h1m24.447908277s) [upload=211178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:27:30.016832424 (203h1m24.447912555s) [upload=211178]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:27:30.463480034 (203h1m24.894560165s) [seg_pos={0 0}, upload=211178]: Begin upload segment... Aug 17 04:27:30.463526713 (203h1m24.894606844s) [seg_pos={0 0}, upload=211178]: Done begin upload segment. Aug 17 04:27:30.463547864 (203h1m24.894627995s) [noise=true , address=77.90.31.12:4610 , piece_id=5HQU72EXU3FU6ZQNTDQDEXWYPATMLKYKEEMGSKNTPAI4JCZAJ5IQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.463559155 (203h1m24.894639286s) [noise=true , address=104.36.82.8:4992 , piece_id=PA7YXTWXXC5SL3UUEWAF7BAJCHDOKZ3BDGZIASS62342EOWUH7IQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.463911099 (203h1m24.89499122s) [noise=true , address=77.90.31.15:4711 , piece_id=CVYW4ML4GFE5IC5SV4G3VEKI47OM5L5O4D7VWKPTUATR2MDIDHBQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.463926858 (203h1m24.895006989s) [noise=true , address=104.36.82.22:4514 , piece_id=CI5POH6B53BJVI6ON4SR3OZSIH33SUFDRA5CJL25CLHLUDZWAS2A, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.464210091 (203h1m24.895290222s) [noise=true , address=77.90.31.14:4678 , piece_id=O3WNM7XRGUZF2WE64X5CCXX5CASWNXVX453NN2BQVDYPZURIOWXA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.464478404 (203h1m24.895558536s) [noise=true , address=104.36.82.60:4369 , piece_id=SZVCNKEW3RAMIAV3XKLIPHYG5YCSTOV2P76NCMS2A2VBMP6R6DBQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.464940458 (203h1m24.896020589s) [noise=true , address=79.127.201.215:4358 , piece_id=RIAIZRP4SCLM2DCKL2QDNHDQEZXURQS5SKMVCUUMHDBXLO7DGJTA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465180658 (203h1m24.896260779s) [noise=true , address=77.90.31.11:4578 , piece_id=PFYRUL44F7FGFRCMUNKVVK4AFP4CSX7Y55Y6O54UCRKNUHWZ7OIQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465394196 (203h1m24.896474327s) [noise=true , address=77.90.31.21:4899 , piece_id=JD3QWQLDB55IJIQ7WUWC3GFTQHRF4UBFZMBGQO4BC7DIMIVSM3KA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465601532 (203h1m24.896681664s) [noise=true , address=104.36.82.2:4762 , piece_id=5MFQVM2XDPNGK4JVFUS7GXCA3M5FDPMG4VHVVXDQ6KLL3F5VSEBA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465820732 (203h1m24.896900853s) [noise=true , address=104.36.82.23:4466 , piece_id=TTBPYPOV6G6VPYTH3BZBA5I2FVW4ORLGAE34JSE6IP4W3UOR6ERQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466052235 (203h1m24.897132366s) [noise=true , address=104.36.82.34:4549 , piece_id=EUAUXMTSRI4UNN4T6XBFL5GXSAP4W4WJ2Z6YBEOGLWLXBKGDYF4Q, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466300020 (203h1m24.897380151s) [noise=true , address=104.36.82.3:4776 , piece_id=ZGFMY6UVQIGXG4XZBWLKMJRWGHMV4ONJ7BBTBXUQOOSTZL7MGK3Q, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466559126 (203h1m24.897639257s) [noise=true , address=104.36.82.5:4872 , piece_id=QDPEEVMTYH35DAERQVGB55IBSNG7NEZCMFNN2GS5JPLGDC4L54IQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466765811 (203h1m24.897845942s) [noise=true , address=79.127.205.233:4646 , piece_id=PQVSNUKLSQ3Q7VLDAGOST7ZIDT2BHVBS43IUZ4X3V7NVFYE67UNA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466965002 (203h1m24.898045123s) [noise=true , address=77.90.31.19:4860 , piece_id=54C73ZPECTG4GPKYMTVOJVOIRRLF6XNE3F36FK632WODFNRBJVXA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.467163783 (203h1m24.898243914s) [noise=true , address=45.91.116.7:4018 , piece_id=GEKZFUTPVL6CJHTRK25E5EL3KF4Y2LASY6CLKA2G5YNFAWFGXBQQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.467361531 (203h1m24.898441662s) [noise=true , address=104.36.82.38:4714 , piece_id=V247PLY3PWQLOGIV6EPXFJDTEI6A2WKSV6HUDNDB4N7ZQSJ7HFDA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:31.608869595 (203h1m26.039949716s) [upload=211178]: Next returned nil segment Aug 17 04:27:31.608891085 (203h1m26.039971266s) [upload=211178]: Waiting for error group managing segments... Aug 17 04:27:31.900075977 (203h1m26.331156108s) [noise=true , address=79.127.205.233:4646 , piece_id=PQVSNUKLSQ3Q7VLDAGOST7ZIDT2BHVBS43IUZ4X3V7NVFYE67UNA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.352306149 (203h1m26.78338628s) [noise=true , address=104.36.82.60:4369 , piece_id=SZVCNKEW3RAMIAV3XKLIPHYG5YCSTOV2P76NCMS2A2VBMP6R6DBQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.357051599 (203h1m26.78813172s) [noise=true , address=104.36.82.8:4992 , piece_id=PA7YXTWXXC5SL3UUEWAF7BAJCHDOKZ3BDGZIASS62342EOWUH7IQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.368906113 (203h1m26.799986244s) [noise=true , address=104.36.82.2:4762 , piece_id=5MFQVM2XDPNGK4JVFUS7GXCA3M5FDPMG4VHVVXDQ6KLL3F5VSEBA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.399599454 (203h1m26.830679585s) [noise=true , address=104.36.82.5:4872 , piece_id=QDPEEVMTYH35DAERQVGB55IBSNG7NEZCMFNN2GS5JPLGDC4L54IQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.422356189 (203h1m26.85343632s) [noise=true , address=104.36.82.38:4714 , piece_id=V247PLY3PWQLOGIV6EPXFJDTEI6A2WKSV6HUDNDB4N7ZQSJ7HFDA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.423908750 (203h1m26.854988881s) [noise=true , address=77.90.31.11:4578 , piece_id=PFYRUL44F7FGFRCMUNKVVK4AFP4CSX7Y55Y6O54UCRKNUHWZ7OIQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.440077859 (203h1m26.87115801s) [noise=true , address=104.36.82.22:4514 , piece_id=CI5POH6B53BJVI6ON4SR3OZSIH33SUFDRA5CJL25CLHLUDZWAS2A, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.440882980 (203h1m26.871963111s) [noise=true , address=104.36.82.34:4549 , piece_id=EUAUXMTSRI4UNN4T6XBFL5GXSAP4W4WJ2Z6YBEOGLWLXBKGDYF4Q, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.454743903 (203h1m26.885824034s) [noise=true , address=79.127.201.215:4358 , piece_id=RIAIZRP4SCLM2DCKL2QDNHDQEZXURQS5SKMVCUUMHDBXLO7DGJTA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.464618066 (203h1m26.895698198s) [noise=true , address=77.90.31.14:4678 , piece_id=O3WNM7XRGUZF2WE64X5CCXX5CASWNXVX453NN2BQVDYPZURIOWXA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.468942671 (203h1m26.900022793s) [noise=true , address=45.91.116.7:4018 , piece_id=GEKZFUTPVL6CJHTRK25E5EL3KF4Y2LASY6CLKA2G5YNFAWFGXBQQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.495852825 (203h1m26.926932956s) [noise=true , address=104.36.82.3:4776 , piece_id=ZGFMY6UVQIGXG4XZBWLKMJRWGHMV4ONJ7BBTBXUQOOSTZL7MGK3Q, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.504721685 (203h1m26.935801816s) [noise=true , address=104.36.82.23:4466 , piece_id=TTBPYPOV6G6VPYTH3BZBA5I2FVW4ORLGAE34JSE6IP4W3UOR6ERQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.504743026 (203h1m26.935823147s) [seg_pos={0 0}, upload=211178]: Segment reached optimal threshold of 14 pieces. Aug 17 04:27:32.504932047 (203h1m26.936012178s) [noise=true , address=77.90.31.15:4711 , piece_id=CVYW4ML4GFE5IC5SV4G3VEKI47OM5L5O4D7VWKPTUATR2MDIDHBQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Aug 17 04:27:32.505005558 (203h1m26.936085689s) [noise=true , address=77.90.31.21:4899 , piece_id=JD3QWQLDB55IJIQ7WUWC3GFTQHRF4UBFZMBGQO4BC7DIMIVSM3KA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG): protocol: write tcp 188.166.225.170:55756->77.90.31.21:4899: use of closed network connection; write tcp 188.166.225.170:55756->77.90.31.21:4899: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55756->77.90.31.21:4899: use of closed network connection Aug 17 04:27:32.505020727 (203h1m26.936100869s) [noise=true , address=77.90.31.19:4860 , piece_id=54C73ZPECTG4GPKYMTVOJVOIRRLF6XNE3F36FK632WODFNRBJVXA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF Aug 17 04:27:32.505052127 (203h1m26.936132248s) [noise=true , address=77.90.31.12:4610 , piece_id=5HQU72EXU3FU6ZQNTDQDEXWYPATMLKYKEEMGSKNTPAI4JCZAJ5IQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: write tcp 188.166.225.170:40394->77.90.31.12:4610: use of closed network connection; write tcp 188.166.225.170:40394->77.90.31.12:4610: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40394->77.90.31.12:4610: use of closed network connection Aug 17 04:27:32.505062547 (203h1m26.936142678s) [seg_pos={0 0}, upload=211178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:27:32.505114917 (203h1m26.936195048s) [upload=211178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:27:32.505118574 (203h1m26.936198705s) [upload=211178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:27:32.796719122 (203h1m27.227799253s) [upload=211178]: Done uploading segments... Aug 17 04:27:32.796750893 (203h1m27.227831024s) [upload=211178]: Upload finished. err: <nil> Duration: 5.588249448s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................................XXXX.......] duration: 5.79739524s FAIL (2025-08-17 04:29:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.797420948s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X.........X........................................] duration: 1.877846728s FAIL (2025-08-17 04:29:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 03:56:48.655583790 (202h30m43.086663921s) [upload=210657]: Starting upload Aug 17 03:56:48.655627825 (202h30m43.086707956s) [upload=210657]: Done starting upload Aug 17 03:56:48.655720953 (202h30m43.086801084s) [upload=210657]: Uploading segments... Aug 17 03:56:48.655806216 (202h30m43.086886347s) [upload=210657]: Got next segment. Inline: true Aug 17 03:56:48.655811276 (202h30m43.086891407s) [upload=210657]: Waiting for error group managing segments... Aug 17 03:56:48.655854448 (202h30m43.086934579s) [upload=210657]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 03:56:48.655859227 (202h30m43.086939358s) [upload=210657]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 03:56:48.655861832 (202h30m43.086941963s) [upload=210657]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 03:56:48.869012190 (202h30m43.300092331s) [upload=210657]: Done uploading segments... Aug 17 03:56:48.869031557 (202h30m43.300111708s) [upload=210657]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.877876845s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................................XXXX.......] duration: 6.136115991s FAIL (2025-08-17 04:29:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.13617321s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 6.390224426s FAIL (2025-08-17 04:28:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.390246307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X.........X.......................................] duration: 2.021431564s FAIL (2025-08-17 04:28:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 03:56:48.655583790 (202h30m43.086663921s) [upload=210657]: Starting upload Aug 17 03:56:48.655627825 (202h30m43.086707956s) [upload=210657]: Done starting upload Aug 17 03:56:48.655720953 (202h30m43.086801084s) [upload=210657]: Uploading segments... Aug 17 03:56:48.655806216 (202h30m43.086886347s) [upload=210657]: Got next segment. Inline: true Aug 17 03:56:48.655811276 (202h30m43.086891407s) [upload=210657]: Waiting for error group managing segments... Aug 17 03:56:48.655854448 (202h30m43.086934579s) [upload=210657]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 03:56:48.655859227 (202h30m43.086939358s) [upload=210657]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 03:56:48.655861832 (202h30m43.086941963s) [upload=210657]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 03:56:48.869012190 (202h30m43.300092331s) [upload=210657]: Done uploading segments... Aug 17 03:56:48.869031557 (202h30m43.300111708s) [upload=210657]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.02145985s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................XXX........] duration: 5.64702074s FAIL (2025-08-17 04:28:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.64705222s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................XXXX......] duration: 5.810987344s FAIL (2025-08-17 04:28:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.811002181s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [...............X.X......................X..........XXXX.X.X.] duration: 5.799515291s FAIL (2025-08-17 04:28:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 17 04:27:30.016543050 (203h1m24.447623181s) [upload=211178]: Starting upload Aug 17 04:27:30.016583197 (203h1m24.447663328s) [upload=211178]: Done starting upload Aug 17 04:27:30.016768071 (203h1m24.447848202s) [upload=211178]: Uploading segments... Aug 17 04:27:30.016798940 (203h1m24.447879071s) [upload=211178]: Got next segment. Inline: false Aug 17 04:27:30.016828156 (203h1m24.447908277s) [upload=211178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:27:30.016832424 (203h1m24.447912555s) [upload=211178]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:27:30.463480034 (203h1m24.894560165s) [seg_pos={0 0}, upload=211178]: Begin upload segment... Aug 17 04:27:30.463526713 (203h1m24.894606844s) [seg_pos={0 0}, upload=211178]: Done begin upload segment. Aug 17 04:27:30.463547864 (203h1m24.894627995s) [noise=true , address=77.90.31.12:4610 , piece_id=5HQU72EXU3FU6ZQNTDQDEXWYPATMLKYKEEMGSKNTPAI4JCZAJ5IQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.463559155 (203h1m24.894639286s) [noise=true , address=104.36.82.8:4992 , piece_id=PA7YXTWXXC5SL3UUEWAF7BAJCHDOKZ3BDGZIASS62342EOWUH7IQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.463911099 (203h1m24.89499122s) [noise=true , address=77.90.31.15:4711 , piece_id=CVYW4ML4GFE5IC5SV4G3VEKI47OM5L5O4D7VWKPTUATR2MDIDHBQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.463926858 (203h1m24.895006989s) [noise=true , address=104.36.82.22:4514 , piece_id=CI5POH6B53BJVI6ON4SR3OZSIH33SUFDRA5CJL25CLHLUDZWAS2A, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.464210091 (203h1m24.895290222s) [noise=true , address=77.90.31.14:4678 , piece_id=O3WNM7XRGUZF2WE64X5CCXX5CASWNXVX453NN2BQVDYPZURIOWXA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.464478404 (203h1m24.895558536s) [noise=true , address=104.36.82.60:4369 , piece_id=SZVCNKEW3RAMIAV3XKLIPHYG5YCSTOV2P76NCMS2A2VBMP6R6DBQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.464940458 (203h1m24.896020589s) [noise=true , address=79.127.201.215:4358 , piece_id=RIAIZRP4SCLM2DCKL2QDNHDQEZXURQS5SKMVCUUMHDBXLO7DGJTA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465180658 (203h1m24.896260779s) [noise=true , address=77.90.31.11:4578 , piece_id=PFYRUL44F7FGFRCMUNKVVK4AFP4CSX7Y55Y6O54UCRKNUHWZ7OIQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465394196 (203h1m24.896474327s) [noise=true , address=77.90.31.21:4899 , piece_id=JD3QWQLDB55IJIQ7WUWC3GFTQHRF4UBFZMBGQO4BC7DIMIVSM3KA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465601532 (203h1m24.896681664s) [noise=true , address=104.36.82.2:4762 , piece_id=5MFQVM2XDPNGK4JVFUS7GXCA3M5FDPMG4VHVVXDQ6KLL3F5VSEBA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465820732 (203h1m24.896900853s) [noise=true , address=104.36.82.23:4466 , piece_id=TTBPYPOV6G6VPYTH3BZBA5I2FVW4ORLGAE34JSE6IP4W3UOR6ERQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466052235 (203h1m24.897132366s) [noise=true , address=104.36.82.34:4549 , piece_id=EUAUXMTSRI4UNN4T6XBFL5GXSAP4W4WJ2Z6YBEOGLWLXBKGDYF4Q, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466300020 (203h1m24.897380151s) [noise=true , address=104.36.82.3:4776 , piece_id=ZGFMY6UVQIGXG4XZBWLKMJRWGHMV4ONJ7BBTBXUQOOSTZL7MGK3Q, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466559126 (203h1m24.897639257s) [noise=true , address=104.36.82.5:4872 , piece_id=QDPEEVMTYH35DAERQVGB55IBSNG7NEZCMFNN2GS5JPLGDC4L54IQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466765811 (203h1m24.897845942s) [noise=true , address=79.127.205.233:4646 , piece_id=PQVSNUKLSQ3Q7VLDAGOST7ZIDT2BHVBS43IUZ4X3V7NVFYE67UNA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466965002 (203h1m24.898045123s) [noise=true , address=77.90.31.19:4860 , piece_id=54C73ZPECTG4GPKYMTVOJVOIRRLF6XNE3F36FK632WODFNRBJVXA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.467163783 (203h1m24.898243914s) [noise=true , address=45.91.116.7:4018 , piece_id=GEKZFUTPVL6CJHTRK25E5EL3KF4Y2LASY6CLKA2G5YNFAWFGXBQQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.467361531 (203h1m24.898441662s) [noise=true , address=104.36.82.38:4714 , piece_id=V247PLY3PWQLOGIV6EPXFJDTEI6A2WKSV6HUDNDB4N7ZQSJ7HFDA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:31.608869595 (203h1m26.039949716s) [upload=211178]: Next returned nil segment Aug 17 04:27:31.608891085 (203h1m26.039971266s) [upload=211178]: Waiting for error group managing segments... Aug 17 04:27:31.900075977 (203h1m26.331156108s) [noise=true , address=79.127.205.233:4646 , piece_id=PQVSNUKLSQ3Q7VLDAGOST7ZIDT2BHVBS43IUZ4X3V7NVFYE67UNA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.352306149 (203h1m26.78338628s) [noise=true , address=104.36.82.60:4369 , piece_id=SZVCNKEW3RAMIAV3XKLIPHYG5YCSTOV2P76NCMS2A2VBMP6R6DBQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.357051599 (203h1m26.78813172s) [noise=true , address=104.36.82.8:4992 , piece_id=PA7YXTWXXC5SL3UUEWAF7BAJCHDOKZ3BDGZIASS62342EOWUH7IQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.368906113 (203h1m26.799986244s) [noise=true , address=104.36.82.2:4762 , piece_id=5MFQVM2XDPNGK4JVFUS7GXCA3M5FDPMG4VHVVXDQ6KLL3F5VSEBA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.399599454 (203h1m26.830679585s) [noise=true , address=104.36.82.5:4872 , piece_id=QDPEEVMTYH35DAERQVGB55IBSNG7NEZCMFNN2GS5JPLGDC4L54IQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.422356189 (203h1m26.85343632s) [noise=true , address=104.36.82.38:4714 , piece_id=V247PLY3PWQLOGIV6EPXFJDTEI6A2WKSV6HUDNDB4N7ZQSJ7HFDA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.423908750 (203h1m26.854988881s) [noise=true , address=77.90.31.11:4578 , piece_id=PFYRUL44F7FGFRCMUNKVVK4AFP4CSX7Y55Y6O54UCRKNUHWZ7OIQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.440077859 (203h1m26.87115801s) [noise=true , address=104.36.82.22:4514 , piece_id=CI5POH6B53BJVI6ON4SR3OZSIH33SUFDRA5CJL25CLHLUDZWAS2A, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.440882980 (203h1m26.871963111s) [noise=true , address=104.36.82.34:4549 , piece_id=EUAUXMTSRI4UNN4T6XBFL5GXSAP4W4WJ2Z6YBEOGLWLXBKGDYF4Q, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.454743903 (203h1m26.885824034s) [noise=true , address=79.127.201.215:4358 , piece_id=RIAIZRP4SCLM2DCKL2QDNHDQEZXURQS5SKMVCUUMHDBXLO7DGJTA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.464618066 (203h1m26.895698198s) [noise=true , address=77.90.31.14:4678 , piece_id=O3WNM7XRGUZF2WE64X5CCXX5CASWNXVX453NN2BQVDYPZURIOWXA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.468942671 (203h1m26.900022793s) [noise=true , address=45.91.116.7:4018 , piece_id=GEKZFUTPVL6CJHTRK25E5EL3KF4Y2LASY6CLKA2G5YNFAWFGXBQQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.495852825 (203h1m26.926932956s) [noise=true , address=104.36.82.3:4776 , piece_id=ZGFMY6UVQIGXG4XZBWLKMJRWGHMV4ONJ7BBTBXUQOOSTZL7MGK3Q, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.504721685 (203h1m26.935801816s) [noise=true , address=104.36.82.23:4466 , piece_id=TTBPYPOV6G6VPYTH3BZBA5I2FVW4ORLGAE34JSE6IP4W3UOR6ERQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.504743026 (203h1m26.935823147s) [seg_pos={0 0}, upload=211178]: Segment reached optimal threshold of 14 pieces. Aug 17 04:27:32.504932047 (203h1m26.936012178s) [noise=true , address=77.90.31.15:4711 , piece_id=CVYW4ML4GFE5IC5SV4G3VEKI47OM5L5O4D7VWKPTUATR2MDIDHBQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Aug 17 04:27:32.505005558 (203h1m26.936085689s) [noise=true , address=77.90.31.21:4899 , piece_id=JD3QWQLDB55IJIQ7WUWC3GFTQHRF4UBFZMBGQO4BC7DIMIVSM3KA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG): protocol: write tcp 188.166.225.170:55756->77.90.31.21:4899: use of closed network connection; write tcp 188.166.225.170:55756->77.90.31.21:4899: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55756->77.90.31.21:4899: use of closed network connection Aug 17 04:27:32.505020727 (203h1m26.936100869s) [noise=true , address=77.90.31.19:4860 , piece_id=54C73ZPECTG4GPKYMTVOJVOIRRLF6XNE3F36FK632WODFNRBJVXA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF Aug 17 04:27:32.505052127 (203h1m26.936132248s) [noise=true , address=77.90.31.12:4610 , piece_id=5HQU72EXU3FU6ZQNTDQDEXWYPATMLKYKEEMGSKNTPAI4JCZAJ5IQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: write tcp 188.166.225.170:40394->77.90.31.12:4610: use of closed network connection; write tcp 188.166.225.170:40394->77.90.31.12:4610: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40394->77.90.31.12:4610: use of closed network connection Aug 17 04:27:32.505062547 (203h1m26.936142678s) [seg_pos={0 0}, upload=211178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:27:32.505114917 (203h1m26.936195048s) [upload=211178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:27:32.505118574 (203h1m26.936198705s) [upload=211178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:27:32.796719122 (203h1m27.227799253s) [upload=211178]: Done uploading segments... Aug 17 04:27:32.796750893 (203h1m27.227831024s) [upload=211178]: Upload finished. err: <nil> Duration: 5.799534779s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 6.423118524s FAIL (2025-08-17 04:28:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.423143693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................XXXX......] duration: 5.744985981s FAIL (2025-08-17 04:28:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.745013093s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X.........X......................................] duration: 1.844338789s FAIL (2025-08-17 04:28:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 03:56:48.655583790 (202h30m43.086663921s) [upload=210657]: Starting upload Aug 17 03:56:48.655627825 (202h30m43.086707956s) [upload=210657]: Done starting upload Aug 17 03:56:48.655720953 (202h30m43.086801084s) [upload=210657]: Uploading segments... Aug 17 03:56:48.655806216 (202h30m43.086886347s) [upload=210657]: Got next segment. Inline: true Aug 17 03:56:48.655811276 (202h30m43.086891407s) [upload=210657]: Waiting for error group managing segments... Aug 17 03:56:48.655854448 (202h30m43.086934579s) [upload=210657]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 03:56:48.655859227 (202h30m43.086939358s) [upload=210657]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 03:56:48.655861832 (202h30m43.086941963s) [upload=210657]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 03:56:48.869012190 (202h30m43.300092331s) [upload=210657]: Done uploading segments... Aug 17 03:56:48.869031557 (202h30m43.300111708s) [upload=210657]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.844356394s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................XXX.......] duration: 5.783411286s FAIL (2025-08-17 04:28:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.783431674s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................XXXX.....] duration: 5.904041697s FAIL (2025-08-17 04:27:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.904073789s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [................X.X......................X..........XXXX.X.X] duration: 4.103922234s FAIL (2025-08-17 04:27:33 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 17 04:27:30.016543050 (203h1m24.447623181s) [upload=211178]: Starting upload Aug 17 04:27:30.016583197 (203h1m24.447663328s) [upload=211178]: Done starting upload Aug 17 04:27:30.016768071 (203h1m24.447848202s) [upload=211178]: Uploading segments... Aug 17 04:27:30.016798940 (203h1m24.447879071s) [upload=211178]: Got next segment. Inline: false Aug 17 04:27:30.016828156 (203h1m24.447908277s) [upload=211178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:27:30.016832424 (203h1m24.447912555s) [upload=211178]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:27:30.463480034 (203h1m24.894560165s) [seg_pos={0 0}, upload=211178]: Begin upload segment... Aug 17 04:27:30.463526713 (203h1m24.894606844s) [seg_pos={0 0}, upload=211178]: Done begin upload segment. Aug 17 04:27:30.463547864 (203h1m24.894627995s) [noise=true , address=77.90.31.12:4610 , piece_id=5HQU72EXU3FU6ZQNTDQDEXWYPATMLKYKEEMGSKNTPAI4JCZAJ5IQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.463559155 (203h1m24.894639286s) [noise=true , address=104.36.82.8:4992 , piece_id=PA7YXTWXXC5SL3UUEWAF7BAJCHDOKZ3BDGZIASS62342EOWUH7IQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.463911099 (203h1m24.89499122s) [noise=true , address=77.90.31.15:4711 , piece_id=CVYW4ML4GFE5IC5SV4G3VEKI47OM5L5O4D7VWKPTUATR2MDIDHBQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.463926858 (203h1m24.895006989s) [noise=true , address=104.36.82.22:4514 , piece_id=CI5POH6B53BJVI6ON4SR3OZSIH33SUFDRA5CJL25CLHLUDZWAS2A, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.464210091 (203h1m24.895290222s) [noise=true , address=77.90.31.14:4678 , piece_id=O3WNM7XRGUZF2WE64X5CCXX5CASWNXVX453NN2BQVDYPZURIOWXA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.464478404 (203h1m24.895558536s) [noise=true , address=104.36.82.60:4369 , piece_id=SZVCNKEW3RAMIAV3XKLIPHYG5YCSTOV2P76NCMS2A2VBMP6R6DBQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.464940458 (203h1m24.896020589s) [noise=true , address=79.127.201.215:4358 , piece_id=RIAIZRP4SCLM2DCKL2QDNHDQEZXURQS5SKMVCUUMHDBXLO7DGJTA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465180658 (203h1m24.896260779s) [noise=true , address=77.90.31.11:4578 , piece_id=PFYRUL44F7FGFRCMUNKVVK4AFP4CSX7Y55Y6O54UCRKNUHWZ7OIQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465394196 (203h1m24.896474327s) [noise=true , address=77.90.31.21:4899 , piece_id=JD3QWQLDB55IJIQ7WUWC3GFTQHRF4UBFZMBGQO4BC7DIMIVSM3KA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465601532 (203h1m24.896681664s) [noise=true , address=104.36.82.2:4762 , piece_id=5MFQVM2XDPNGK4JVFUS7GXCA3M5FDPMG4VHVVXDQ6KLL3F5VSEBA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.465820732 (203h1m24.896900853s) [noise=true , address=104.36.82.23:4466 , piece_id=TTBPYPOV6G6VPYTH3BZBA5I2FVW4ORLGAE34JSE6IP4W3UOR6ERQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466052235 (203h1m24.897132366s) [noise=true , address=104.36.82.34:4549 , piece_id=EUAUXMTSRI4UNN4T6XBFL5GXSAP4W4WJ2Z6YBEOGLWLXBKGDYF4Q, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466300020 (203h1m24.897380151s) [noise=true , address=104.36.82.3:4776 , piece_id=ZGFMY6UVQIGXG4XZBWLKMJRWGHMV4ONJ7BBTBXUQOOSTZL7MGK3Q, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466559126 (203h1m24.897639257s) [noise=true , address=104.36.82.5:4872 , piece_id=QDPEEVMTYH35DAERQVGB55IBSNG7NEZCMFNN2GS5JPLGDC4L54IQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466765811 (203h1m24.897845942s) [noise=true , address=79.127.205.233:4646 , piece_id=PQVSNUKLSQ3Q7VLDAGOST7ZIDT2BHVBS43IUZ4X3V7NVFYE67UNA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.466965002 (203h1m24.898045123s) [noise=true , address=77.90.31.19:4860 , piece_id=54C73ZPECTG4GPKYMTVOJVOIRRLF6XNE3F36FK632WODFNRBJVXA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.467163783 (203h1m24.898243914s) [noise=true , address=45.91.116.7:4018 , piece_id=GEKZFUTPVL6CJHTRK25E5EL3KF4Y2LASY6CLKA2G5YNFAWFGXBQQ, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:30.467361531 (203h1m24.898441662s) [noise=true , address=104.36.82.38:4714 , piece_id=V247PLY3PWQLOGIV6EPXFJDTEI6A2WKSV6HUDNDB4N7ZQSJ7HFDA, seg_pos={0 0}, upload=211178]: Uploading piece... Aug 17 04:27:31.608869595 (203h1m26.039949716s) [upload=211178]: Next returned nil segment Aug 17 04:27:31.608891085 (203h1m26.039971266s) [upload=211178]: Waiting for error group managing segments... Aug 17 04:27:31.900075977 (203h1m26.331156108s) [noise=true , address=79.127.205.233:4646 , piece_id=PQVSNUKLSQ3Q7VLDAGOST7ZIDT2BHVBS43IUZ4X3V7NVFYE67UNA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.352306149 (203h1m26.78338628s) [noise=true , address=104.36.82.60:4369 , piece_id=SZVCNKEW3RAMIAV3XKLIPHYG5YCSTOV2P76NCMS2A2VBMP6R6DBQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.357051599 (203h1m26.78813172s) [noise=true , address=104.36.82.8:4992 , piece_id=PA7YXTWXXC5SL3UUEWAF7BAJCHDOKZ3BDGZIASS62342EOWUH7IQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.368906113 (203h1m26.799986244s) [noise=true , address=104.36.82.2:4762 , piece_id=5MFQVM2XDPNGK4JVFUS7GXCA3M5FDPMG4VHVVXDQ6KLL3F5VSEBA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.399599454 (203h1m26.830679585s) [noise=true , address=104.36.82.5:4872 , piece_id=QDPEEVMTYH35DAERQVGB55IBSNG7NEZCMFNN2GS5JPLGDC4L54IQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.422356189 (203h1m26.85343632s) [noise=true , address=104.36.82.38:4714 , piece_id=V247PLY3PWQLOGIV6EPXFJDTEI6A2WKSV6HUDNDB4N7ZQSJ7HFDA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.423908750 (203h1m26.854988881s) [noise=true , address=77.90.31.11:4578 , piece_id=PFYRUL44F7FGFRCMUNKVVK4AFP4CSX7Y55Y6O54UCRKNUHWZ7OIQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.440077859 (203h1m26.87115801s) [noise=true , address=104.36.82.22:4514 , piece_id=CI5POH6B53BJVI6ON4SR3OZSIH33SUFDRA5CJL25CLHLUDZWAS2A, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.440882980 (203h1m26.871963111s) [noise=true , address=104.36.82.34:4549 , piece_id=EUAUXMTSRI4UNN4T6XBFL5GXSAP4W4WJ2Z6YBEOGLWLXBKGDYF4Q, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.454743903 (203h1m26.885824034s) [noise=true , address=79.127.201.215:4358 , piece_id=RIAIZRP4SCLM2DCKL2QDNHDQEZXURQS5SKMVCUUMHDBXLO7DGJTA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.464618066 (203h1m26.895698198s) [noise=true , address=77.90.31.14:4678 , piece_id=O3WNM7XRGUZF2WE64X5CCXX5CASWNXVX453NN2BQVDYPZURIOWXA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.468942671 (203h1m26.900022793s) [noise=true , address=45.91.116.7:4018 , piece_id=GEKZFUTPVL6CJHTRK25E5EL3KF4Y2LASY6CLKA2G5YNFAWFGXBQQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.495852825 (203h1m26.926932956s) [noise=true , address=104.36.82.3:4776 , piece_id=ZGFMY6UVQIGXG4XZBWLKMJRWGHMV4ONJ7BBTBXUQOOSTZL7MGK3Q, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.504721685 (203h1m26.935801816s) [noise=true , address=104.36.82.23:4466 , piece_id=TTBPYPOV6G6VPYTH3BZBA5I2FVW4ORLGAE34JSE6IP4W3UOR6ERQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: <nil> Aug 17 04:27:32.504743026 (203h1m26.935823147s) [seg_pos={0 0}, upload=211178]: Segment reached optimal threshold of 14 pieces. Aug 17 04:27:32.504932047 (203h1m26.936012178s) [noise=true , address=77.90.31.15:4711 , piece_id=CVYW4ML4GFE5IC5SV4G3VEKI47OM5L5O4D7VWKPTUATR2MDIDHBQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Aug 17 04:27:32.505005558 (203h1m26.936085689s) [noise=true , address=77.90.31.21:4899 , piece_id=JD3QWQLDB55IJIQ7WUWC3GFTQHRF4UBFZMBGQO4BC7DIMIVSM3KA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG): protocol: write tcp 188.166.225.170:55756->77.90.31.21:4899: use of closed network connection; write tcp 188.166.225.170:55756->77.90.31.21:4899: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55756->77.90.31.21:4899: use of closed network connection Aug 17 04:27:32.505020727 (203h1m26.936100869s) [noise=true , address=77.90.31.19:4860 , piece_id=54C73ZPECTG4GPKYMTVOJVOIRRLF6XNE3F36FK632WODFNRBJVXA, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd): protocol: expected piece hash; context canceled; EOF Aug 17 04:27:32.505052127 (203h1m26.936132248s) [noise=true , address=77.90.31.12:4610 , piece_id=5HQU72EXU3FU6ZQNTDQDEXWYPATMLKYKEEMGSKNTPAI4JCZAJ5IQ, seg_pos={0 0}, upload=211178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: write tcp 188.166.225.170:40394->77.90.31.12:4610: use of closed network connection; write tcp 188.166.225.170:40394->77.90.31.12:4610: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40394->77.90.31.12:4610: use of closed network connection Aug 17 04:27:32.505062547 (203h1m26.936142678s) [seg_pos={0 0}, upload=211178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:27:32.505114917 (203h1m26.936195048s) [upload=211178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:27:32.505118574 (203h1m26.936198705s) [upload=211178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:27:32.796719122 (203h1m27.227799253s) [upload=211178]: Done uploading segments... Aug 17 04:27:32.796750893 (203h1m27.227831024s) [upload=211178]: Upload finished. err: <nil> Duration: 4.103946862s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................XXX......] duration: 5.659234008s FAIL (2025-08-17 04:27:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.659259957s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................XXXX.....] duration: 6.332847836s FAIL (2025-08-17 04:27:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.332870258s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 6.710094746s FAIL (2025-08-17 04:27:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.710118461s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............X.........X.....................................] duration: 1.945879711s FAIL (2025-08-17 04:27:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 03:56:48.655583790 (202h30m43.086663921s) [upload=210657]: Starting upload Aug 17 03:56:48.655627825 (202h30m43.086707956s) [upload=210657]: Done starting upload Aug 17 03:56:48.655720953 (202h30m43.086801084s) [upload=210657]: Uploading segments... Aug 17 03:56:48.655806216 (202h30m43.086886347s) [upload=210657]: Got next segment. Inline: true Aug 17 03:56:48.655811276 (202h30m43.086891407s) [upload=210657]: Waiting for error group managing segments... Aug 17 03:56:48.655854448 (202h30m43.086934579s) [upload=210657]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 03:56:48.655859227 (202h30m43.086939358s) [upload=210657]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 03:56:48.655861832 (202h30m43.086941963s) [upload=210657]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 03:56:48.869012190 (202h30m43.300092331s) [upload=210657]: Done uploading segments... Aug 17 03:56:48.869031557 (202h30m43.300111708s) [upload=210657]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.945904958s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................XXXX....] duration: 5.695433158s FAIL (2025-08-17 04:26:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.695459038s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X.X......................X..........XXXX.X.] duration: 5.604799022s FAIL (2025-08-17 04:26:36 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 17 04:25:18.803895869 (202h59m13.234975991s) [upload=211147]: Starting upload Aug 17 04:25:18.803941467 (202h59m13.235021588s) [upload=211147]: Done starting upload Aug 17 04:25:18.804012804 (202h59m13.235092925s) [upload=211147]: Uploading segments... Aug 17 04:25:18.804107004 (202h59m13.235187125s) [upload=211147]: Got next segment. Inline: false Aug 17 04:25:18.804166277 (202h59m13.235246398s) [upload=211147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:25:18.804170996 (202h59m13.235251117s) [upload=211147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:25:19.256074323 (202h59m13.687154454s) [seg_pos={0 0}, upload=211147]: Begin upload segment... Aug 17 04:25:19.256124990 (202h59m13.687205121s) [seg_pos={0 0}, upload=211147]: Done begin upload segment. Aug 17 04:25:19.256182300 (202h59m13.687262431s) [noise=true , address=77.90.31.21:4906 , piece_id=VEMKB6SJRYM4YQGQU6X2QUV26KWM5SPH3O4PIA5644L2DOKER44A, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.256187971 (202h59m13.687268102s) [noise=true , address=79.127.205.246:4288 , piece_id=FMR3CZOSAQ7G5F7LQLWUWVLRPJMHBZT6PUCZR5SX5OGWHLYBEBNQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.256221264 (202h59m13.687301395s) [noise=true , address=104.36.82.2:4747 , piece_id=GYBRNYM4H7C6JAXGNHC2WDMSHHQL5GKU7GGO5A2AE3W5LIS2Q66A, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.256286660 (202h59m13.687366801s) [noise=true , address=104.36.82.23:4448 , piece_id=YJSMK4APTPM3U5IKC4KTHKT5KYYKM7633GEJHVUAU24PQETDEDLQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.256549061 (202h59m13.687629192s) [noise=true , address=104.36.82.4:4809 , piece_id=LI7EBH7M3VLKYHPE5CEYFUWIK6CKYYG2U4PHOKW4ATR5L7EMJIZQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.256831552 (202h59m13.687911674s) [noise=true , address=45.91.116.9:4274 , piece_id=OY5TQJ4MI6E7XIZQ4HOUY5C7IHGPARSLQ32LPHCXAGJZROW47ECQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.257184478 (202h59m13.688264609s) [noise=true , address=104.36.82.19:4417 , piece_id=ZXE54KX56YDUPAOTLSUQMGDLSM2YQI6Y5JZVWEAPAE57VZTT4VPQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.257470234 (202h59m13.688550365s) [noise=true , address=104.36.82.8:4002 , piece_id=7GJUMOFAMBYOQVMRUSZWQGTIQCIPMOPPUTQXOGFJTXGTQTA4PH7A, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.257695936 (202h59m13.688776067s) [noise=true , address=66.23.193.175:4560 , piece_id=3Y2GZQ74XTI7YOAWBZAHQA2GYZEKJ3QQ4KZJSCZROIRDTOEW7WAQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.257944782 (202h59m13.689024913s) [noise=true , address=79.127.201.215:4360 , piece_id=BK7QMVSNWOQTZ26FWYQW3EGWQIKUS3CVJQROYMLPNQ3JILI3VEXQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.258172729 (202h59m13.68925285s) [noise=true , address=104.36.82.60:4351 , piece_id=KWRP2EGVH7IQQNXXATIQYHBK4D4B63TW6IA5JEC2MQAQSQ2LBK7A, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.258381458 (202h59m13.689461589s) [noise=true , address=104.36.82.22:4521 , piece_id=FQZNT2YN6SMDTIAMAE4JZIKOGURYGR2VLOQZQSDM7BM4EZGHDUJA, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.259481533 (202h59m13.690561674s) [noise=true , address=77.90.31.11:4581 , piece_id=QQ33Q2UNFN6JRDVI3DT37QTRDRDOABSMBZ4EZOHSMR7NSPWIJKLA, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.259649745 (202h59m13.690729876s) [noise=true , address=77.90.31.12:4612 , piece_id=IFO7YV63S7MTGJ6ZJUXN4WVWORF2JSEVNMQWZWKHKD7RKXKU3XZQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.259903510 (202h59m13.690983641s) [noise=true , address=79.127.205.234:4595 , piece_id=Y2D7D4KTXVGXVPBSXZG4DO4NFZN3TLX7BT7QKOU7MQ5D42JB4ASQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.260116057 (202h59m13.691196188s) [noise=true , address=104.36.82.5:4850 , piece_id=KCA7SZHBBSWX3WS6Z665Q4X4LOCATQTBEIJME4LUISRX2V6NQVDA, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.260345676 (202h59m13.691425807s) [noise=true , address=45.91.116.11:4125 , piece_id=PVJZPHWOQULGNJR75EBIJZTALF5MIUPTFGHEHGYWJZXKT5FWBGDQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.260537303 (202h59m13.691617434s) [noise=true , address=104.36.82.39:4744 , piece_id=BDUKZ7FVI6PG3SYEOTZSYPZG7OFK6KUTDM4RBHLMA3LLZZIINGEA, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:20.357319624 (202h59m14.788399755s) [upload=211147]: Next returned nil segment Aug 17 04:25:20.357339180 (202h59m14.788419311s) [upload=211147]: Waiting for error group managing segments... Aug 17 04:25:20.653585507 (202h59m15.084665638s) [noise=true , address=66.23.193.175:4560 , piece_id=3Y2GZQ74XTI7YOAWBZAHQA2GYZEKJ3QQ4KZJSCZROIRDTOEW7WAQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:20.689674213 (202h59m15.120754344s) [noise=true , address=79.127.205.246:4288 , piece_id=FMR3CZOSAQ7G5F7LQLWUWVLRPJMHBZT6PUCZR5SX5OGWHLYBEBNQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:20.690695738 (202h59m15.121775869s) [noise=true , address=79.127.205.234:4595 , piece_id=Y2D7D4KTXVGXVPBSXZG4DO4NFZN3TLX7BT7QKOU7MQ5D42JB4ASQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.141569895 (202h59m15.572650026s) [noise=true , address=104.36.82.2:4747 , piece_id=GYBRNYM4H7C6JAXGNHC2WDMSHHQL5GKU7GGO5A2AE3W5LIS2Q66A, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.152849558 (202h59m15.583929689s) [noise=true , address=104.36.82.60:4351 , piece_id=KWRP2EGVH7IQQNXXATIQYHBK4D4B63TW6IA5JEC2MQAQSQ2LBK7A, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.160872501 (202h59m15.591952632s) [noise=true , address=77.90.31.11:4581 , piece_id=QQ33Q2UNFN6JRDVI3DT37QTRDRDOABSMBZ4EZOHSMR7NSPWIJKLA, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.168839906 (202h59m15.599920027s) [noise=true , address=45.91.116.9:4274 , piece_id=OY5TQJ4MI6E7XIZQ4HOUY5C7IHGPARSLQ32LPHCXAGJZROW47ECQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.175493187 (202h59m15.606573308s) [noise=true , address=45.91.116.11:4125 , piece_id=PVJZPHWOQULGNJR75EBIJZTALF5MIUPTFGHEHGYWJZXKT5FWBGDQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.247095598 (202h59m15.678175729s) [noise=true , address=104.36.82.23:4448 , piece_id=YJSMK4APTPM3U5IKC4KTHKT5KYYKM7633GEJHVUAU24PQETDEDLQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.259353243 (202h59m15.690433364s) [noise=true , address=77.90.31.12:4612 , piece_id=IFO7YV63S7MTGJ6ZJUXN4WVWORF2JSEVNMQWZWKHKD7RKXKU3XZQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.260493414 (202h59m15.691573535s) [noise=true , address=104.36.82.5:4850 , piece_id=KCA7SZHBBSWX3WS6Z665Q4X4LOCATQTBEIJME4LUISRX2V6NQVDA, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.279062076 (202h59m15.710142207s) [noise=true , address=79.127.201.215:4360 , piece_id=BK7QMVSNWOQTZ26FWYQW3EGWQIKUS3CVJQROYMLPNQ3JILI3VEXQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.289320425 (202h59m15.720400556s) [noise=true , address=104.36.82.39:4744 , piece_id=BDUKZ7FVI6PG3SYEOTZSYPZG7OFK6KUTDM4RBHLMA3LLZZIINGEA, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.294425023 (202h59m15.725505154s) [noise=true , address=104.36.82.19:4417 , piece_id=ZXE54KX56YDUPAOTLSUQMGDLSM2YQI6Y5JZVWEAPAE57VZTT4VPQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.294447696 (202h59m15.725527827s) [seg_pos={0 0}, upload=211147]: Segment reached optimal threshold of 14 pieces. Aug 17 04:25:21.294546676 (202h59m15.725626807s) [noise=true , address=104.36.82.4:4809 , piece_id=LI7EBH7M3VLKYHPE5CEYFUWIK6CKYYG2U4PHOKW4ATR5L7EMJIZQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Aug 17 04:25:21.294603755 (202h59m15.725683886s) [noise=true , address=104.36.82.8:4002 , piece_id=7GJUMOFAMBYOQVMRUSZWQGTIQCIPMOPPUTQXOGFJTXGTQTA4PH7A, seg_pos={0 0}, upload=211147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DW1BZ57L7XPH8kTjCqKxBc492jeqwb7eKni3zYjVetMBrmyJ9): protocol: expected piece hash; context canceled; EOF Aug 17 04:25:21.294665273 (202h59m15.725745404s) [noise=true , address=77.90.31.21:4906 , piece_id=VEMKB6SJRYM4YQGQU6X2QUV26KWM5SPH3O4PIA5644L2DOKER44A, seg_pos={0 0}, upload=211147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9): protocol: expected piece hash; context canceled; EOF Aug 17 04:25:21.294700730 (202h59m15.725780861s) [noise=true , address=104.36.82.22:4521 , piece_id=FQZNT2YN6SMDTIAMAE4JZIKOGURYGR2VLOQZQSDM7BM4EZGHDUJA, seg_pos={0 0}, upload=211147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zWiwpaV58h8YKsAgw24fUWHBBYYyqF4UX6GJV316Uxo9Y7gt1): protocol: expected piece hash; context canceled; EOF Aug 17 04:25:21.294736478 (202h59m15.725816609s) [seg_pos={0 0}, upload=211147]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:25:21.294814488 (202h59m15.725894619s) [upload=211147]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:25:21.294818976 (202h59m15.725899107s) [upload=211147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:25:21.623682326 (202h59m16.054762457s) [upload=211147]: Done uploading segments... Aug 17 04:25:21.623701834 (202h59m16.054781965s) [upload=211147]: Upload finished. err: <nil> Duration: 5.604837315s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................XXXX....] duration: 6.449353563s FAIL (2025-08-17 04:26:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.44936785s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............X.........X....................................] duration: 2.018839305s FAIL (2025-08-17 04:26:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 03:56:48.655583790 (202h30m43.086663921s) [upload=210657]: Starting upload Aug 17 03:56:48.655627825 (202h30m43.086707956s) [upload=210657]: Done starting upload Aug 17 03:56:48.655720953 (202h30m43.086801084s) [upload=210657]: Uploading segments... Aug 17 03:56:48.655806216 (202h30m43.086886347s) [upload=210657]: Got next segment. Inline: true Aug 17 03:56:48.655811276 (202h30m43.086891407s) [upload=210657]: Waiting for error group managing segments... Aug 17 03:56:48.655854448 (202h30m43.086934579s) [upload=210657]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 03:56:48.655859227 (202h30m43.086939358s) [upload=210657]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 03:56:48.655861832 (202h30m43.086941963s) [upload=210657]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 03:56:48.869012190 (202h30m43.300092331s) [upload=210657]: Done uploading segments... Aug 17 03:56:48.869031557 (202h30m43.300111708s) [upload=210657]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.018874412s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 6.571775513s FAIL (2025-08-17 04:26:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.571842001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................XXX.....] duration: 6.07894537s FAIL (2025-08-17 04:26:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.078972782s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XXXX...] duration: 6.045327807s FAIL (2025-08-17 04:25:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.045348756s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XXXX...] duration: 6.003871584s FAIL (2025-08-17 04:25:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.00390119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................XXX....] duration: 5.708402313s FAIL (2025-08-17 04:25:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.708427992s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 6.652780879s FAIL (2025-08-17 04:25:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.652810205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X.X......................X..........XXXX.X] duration: 4.211295476s FAIL (2025-08-17 04:25:22 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 17 04:25:18.803895869 (202h59m13.234975991s) [upload=211147]: Starting upload Aug 17 04:25:18.803941467 (202h59m13.235021588s) [upload=211147]: Done starting upload Aug 17 04:25:18.804012804 (202h59m13.235092925s) [upload=211147]: Uploading segments... Aug 17 04:25:18.804107004 (202h59m13.235187125s) [upload=211147]: Got next segment. Inline: false Aug 17 04:25:18.804166277 (202h59m13.235246398s) [upload=211147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:25:18.804170996 (202h59m13.235251117s) [upload=211147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:25:19.256074323 (202h59m13.687154454s) [seg_pos={0 0}, upload=211147]: Begin upload segment... Aug 17 04:25:19.256124990 (202h59m13.687205121s) [seg_pos={0 0}, upload=211147]: Done begin upload segment. Aug 17 04:25:19.256182300 (202h59m13.687262431s) [noise=true , address=77.90.31.21:4906 , piece_id=VEMKB6SJRYM4YQGQU6X2QUV26KWM5SPH3O4PIA5644L2DOKER44A, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.256187971 (202h59m13.687268102s) [noise=true , address=79.127.205.246:4288 , piece_id=FMR3CZOSAQ7G5F7LQLWUWVLRPJMHBZT6PUCZR5SX5OGWHLYBEBNQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.256221264 (202h59m13.687301395s) [noise=true , address=104.36.82.2:4747 , piece_id=GYBRNYM4H7C6JAXGNHC2WDMSHHQL5GKU7GGO5A2AE3W5LIS2Q66A, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.256286660 (202h59m13.687366801s) [noise=true , address=104.36.82.23:4448 , piece_id=YJSMK4APTPM3U5IKC4KTHKT5KYYKM7633GEJHVUAU24PQETDEDLQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.256549061 (202h59m13.687629192s) [noise=true , address=104.36.82.4:4809 , piece_id=LI7EBH7M3VLKYHPE5CEYFUWIK6CKYYG2U4PHOKW4ATR5L7EMJIZQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.256831552 (202h59m13.687911674s) [noise=true , address=45.91.116.9:4274 , piece_id=OY5TQJ4MI6E7XIZQ4HOUY5C7IHGPARSLQ32LPHCXAGJZROW47ECQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.257184478 (202h59m13.688264609s) [noise=true , address=104.36.82.19:4417 , piece_id=ZXE54KX56YDUPAOTLSUQMGDLSM2YQI6Y5JZVWEAPAE57VZTT4VPQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.257470234 (202h59m13.688550365s) [noise=true , address=104.36.82.8:4002 , piece_id=7GJUMOFAMBYOQVMRUSZWQGTIQCIPMOPPUTQXOGFJTXGTQTA4PH7A, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.257695936 (202h59m13.688776067s) [noise=true , address=66.23.193.175:4560 , piece_id=3Y2GZQ74XTI7YOAWBZAHQA2GYZEKJ3QQ4KZJSCZROIRDTOEW7WAQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.257944782 (202h59m13.689024913s) [noise=true , address=79.127.201.215:4360 , piece_id=BK7QMVSNWOQTZ26FWYQW3EGWQIKUS3CVJQROYMLPNQ3JILI3VEXQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.258172729 (202h59m13.68925285s) [noise=true , address=104.36.82.60:4351 , piece_id=KWRP2EGVH7IQQNXXATIQYHBK4D4B63TW6IA5JEC2MQAQSQ2LBK7A, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.258381458 (202h59m13.689461589s) [noise=true , address=104.36.82.22:4521 , piece_id=FQZNT2YN6SMDTIAMAE4JZIKOGURYGR2VLOQZQSDM7BM4EZGHDUJA, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.259481533 (202h59m13.690561674s) [noise=true , address=77.90.31.11:4581 , piece_id=QQ33Q2UNFN6JRDVI3DT37QTRDRDOABSMBZ4EZOHSMR7NSPWIJKLA, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.259649745 (202h59m13.690729876s) [noise=true , address=77.90.31.12:4612 , piece_id=IFO7YV63S7MTGJ6ZJUXN4WVWORF2JSEVNMQWZWKHKD7RKXKU3XZQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.259903510 (202h59m13.690983641s) [noise=true , address=79.127.205.234:4595 , piece_id=Y2D7D4KTXVGXVPBSXZG4DO4NFZN3TLX7BT7QKOU7MQ5D42JB4ASQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.260116057 (202h59m13.691196188s) [noise=true , address=104.36.82.5:4850 , piece_id=KCA7SZHBBSWX3WS6Z665Q4X4LOCATQTBEIJME4LUISRX2V6NQVDA, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.260345676 (202h59m13.691425807s) [noise=true , address=45.91.116.11:4125 , piece_id=PVJZPHWOQULGNJR75EBIJZTALF5MIUPTFGHEHGYWJZXKT5FWBGDQ, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:19.260537303 (202h59m13.691617434s) [noise=true , address=104.36.82.39:4744 , piece_id=BDUKZ7FVI6PG3SYEOTZSYPZG7OFK6KUTDM4RBHLMA3LLZZIINGEA, seg_pos={0 0}, upload=211147]: Uploading piece... Aug 17 04:25:20.357319624 (202h59m14.788399755s) [upload=211147]: Next returned nil segment Aug 17 04:25:20.357339180 (202h59m14.788419311s) [upload=211147]: Waiting for error group managing segments... Aug 17 04:25:20.653585507 (202h59m15.084665638s) [noise=true , address=66.23.193.175:4560 , piece_id=3Y2GZQ74XTI7YOAWBZAHQA2GYZEKJ3QQ4KZJSCZROIRDTOEW7WAQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:20.689674213 (202h59m15.120754344s) [noise=true , address=79.127.205.246:4288 , piece_id=FMR3CZOSAQ7G5F7LQLWUWVLRPJMHBZT6PUCZR5SX5OGWHLYBEBNQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:20.690695738 (202h59m15.121775869s) [noise=true , address=79.127.205.234:4595 , piece_id=Y2D7D4KTXVGXVPBSXZG4DO4NFZN3TLX7BT7QKOU7MQ5D42JB4ASQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.141569895 (202h59m15.572650026s) [noise=true , address=104.36.82.2:4747 , piece_id=GYBRNYM4H7C6JAXGNHC2WDMSHHQL5GKU7GGO5A2AE3W5LIS2Q66A, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.152849558 (202h59m15.583929689s) [noise=true , address=104.36.82.60:4351 , piece_id=KWRP2EGVH7IQQNXXATIQYHBK4D4B63TW6IA5JEC2MQAQSQ2LBK7A, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.160872501 (202h59m15.591952632s) [noise=true , address=77.90.31.11:4581 , piece_id=QQ33Q2UNFN6JRDVI3DT37QTRDRDOABSMBZ4EZOHSMR7NSPWIJKLA, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.168839906 (202h59m15.599920027s) [noise=true , address=45.91.116.9:4274 , piece_id=OY5TQJ4MI6E7XIZQ4HOUY5C7IHGPARSLQ32LPHCXAGJZROW47ECQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.175493187 (202h59m15.606573308s) [noise=true , address=45.91.116.11:4125 , piece_id=PVJZPHWOQULGNJR75EBIJZTALF5MIUPTFGHEHGYWJZXKT5FWBGDQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.247095598 (202h59m15.678175729s) [noise=true , address=104.36.82.23:4448 , piece_id=YJSMK4APTPM3U5IKC4KTHKT5KYYKM7633GEJHVUAU24PQETDEDLQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.259353243 (202h59m15.690433364s) [noise=true , address=77.90.31.12:4612 , piece_id=IFO7YV63S7MTGJ6ZJUXN4WVWORF2JSEVNMQWZWKHKD7RKXKU3XZQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.260493414 (202h59m15.691573535s) [noise=true , address=104.36.82.5:4850 , piece_id=KCA7SZHBBSWX3WS6Z665Q4X4LOCATQTBEIJME4LUISRX2V6NQVDA, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.279062076 (202h59m15.710142207s) [noise=true , address=79.127.201.215:4360 , piece_id=BK7QMVSNWOQTZ26FWYQW3EGWQIKUS3CVJQROYMLPNQ3JILI3VEXQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.289320425 (202h59m15.720400556s) [noise=true , address=104.36.82.39:4744 , piece_id=BDUKZ7FVI6PG3SYEOTZSYPZG7OFK6KUTDM4RBHLMA3LLZZIINGEA, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.294425023 (202h59m15.725505154s) [noise=true , address=104.36.82.19:4417 , piece_id=ZXE54KX56YDUPAOTLSUQMGDLSM2YQI6Y5JZVWEAPAE57VZTT4VPQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: <nil> Aug 17 04:25:21.294447696 (202h59m15.725527827s) [seg_pos={0 0}, upload=211147]: Segment reached optimal threshold of 14 pieces. Aug 17 04:25:21.294546676 (202h59m15.725626807s) [noise=true , address=104.36.82.4:4809 , piece_id=LI7EBH7M3VLKYHPE5CEYFUWIK6CKYYG2U4PHOKW4ATR5L7EMJIZQ, seg_pos={0 0}, upload=211147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Aug 17 04:25:21.294603755 (202h59m15.725683886s) [noise=true , address=104.36.82.8:4002 , piece_id=7GJUMOFAMBYOQVMRUSZWQGTIQCIPMOPPUTQXOGFJTXGTQTA4PH7A, seg_pos={0 0}, upload=211147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DW1BZ57L7XPH8kTjCqKxBc492jeqwb7eKni3zYjVetMBrmyJ9): protocol: expected piece hash; context canceled; EOF Aug 17 04:25:21.294665273 (202h59m15.725745404s) [noise=true , address=77.90.31.21:4906 , piece_id=VEMKB6SJRYM4YQGQU6X2QUV26KWM5SPH3O4PIA5644L2DOKER44A, seg_pos={0 0}, upload=211147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9): protocol: expected piece hash; context canceled; EOF Aug 17 04:25:21.294700730 (202h59m15.725780861s) [noise=true , address=104.36.82.22:4521 , piece_id=FQZNT2YN6SMDTIAMAE4JZIKOGURYGR2VLOQZQSDM7BM4EZGHDUJA, seg_pos={0 0}, upload=211147]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zWiwpaV58h8YKsAgw24fUWHBBYYyqF4UX6GJV316Uxo9Y7gt1): protocol: expected piece hash; context canceled; EOF Aug 17 04:25:21.294736478 (202h59m15.725816609s) [seg_pos={0 0}, upload=211147]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:25:21.294814488 (202h59m15.725894619s) [upload=211147]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:25:21.294818976 (202h59m15.725899107s) [upload=211147]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:25:21.623682326 (202h59m16.054762457s) [upload=211147]: Done uploading segments... Aug 17 04:25:21.623701834 (202h59m16.054781965s) [upload=211147]: Upload finished. err: <nil> Duration: 4.211321244s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................XXX...] duration: 5.826771015s FAIL (2025-08-17 04:25:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.826789841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............X.........X...................................] duration: 2.338639042s FAIL (2025-08-17 04:25:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 03:56:48.655583790 (202h30m43.086663921s) [upload=210657]: Starting upload Aug 17 03:56:48.655627825 (202h30m43.086707956s) [upload=210657]: Done starting upload Aug 17 03:56:48.655720953 (202h30m43.086801084s) [upload=210657]: Uploading segments... Aug 17 03:56:48.655806216 (202h30m43.086886347s) [upload=210657]: Got next segment. Inline: true Aug 17 03:56:48.655811276 (202h30m43.086891407s) [upload=210657]: Waiting for error group managing segments... Aug 17 03:56:48.655854448 (202h30m43.086934579s) [upload=210657]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 03:56:48.655859227 (202h30m43.086939358s) [upload=210657]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 03:56:48.655861832 (202h30m43.086941963s) [upload=210657]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 03:56:48.869012190 (202h30m43.300092331s) [upload=210657]: Done uploading segments... Aug 17 03:56:48.869031557 (202h30m43.300111708s) [upload=210657]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.338665733s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXXX..] duration: 6.288249765s FAIL (2025-08-17 04:24:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.288271445s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXXX..] duration: 6.226062699s FAIL (2025-08-17 04:24:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.226088589s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration: 6.024679574s FAIL (2025-08-17 04:24:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.024696716s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 6.430812655s FAIL (2025-08-17 04:24:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.430833065s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)