Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 17 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXX.X...............................] duration: 1.926056997s FAIL (2025-08-07 22:49:12 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.926086544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................XXX.............................] duration: 5.940502966s FAIL (2025-08-07 22:49:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.940644427s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXX.X..X...........................] duration: 1.926728252s FAIL (2025-08-07 22:49:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.926748812s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......................XXXXXXXXX..X.X.......................] duration: 7.221582979s FAIL (2025-08-07 22:48:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.221611062s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 330.0961ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:48:52 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 330.125356ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXX...X.......................] duration: 6.062215253s FAIL (2025-08-07 22:48:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.062237394s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [....................X..X....................................] duration: 736.226186ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:48:51 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 736.253448ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXX.XX..X..............................] duration: 1.845363709s FAIL (2025-08-07 22:48:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.845393968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 17 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXX.X..............................] duration: 1.887721773s FAIL (2025-08-07 22:48:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.887740487s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 641.736262ms target: https://us1.storj.io/login FAIL (2025-08-07 22:48:38 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 641.770157ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXX.X........................] duration: 5.619262539s FAIL (2025-08-07 22:48:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.619290513s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X..............X.........X....X.................X.....] duration: 7.246499294s FAIL (2025-08-07 22:48:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 81af0bd8b34b2e69389f7db10076d194a3d6fbdd58cd01527d1b66bf7a00fe45 != before 7800c7ceda312caa58fa18fda98736593dbd42db38936a6dbdeb420eade030ff ======LOGS====== pload=13556]: Done starting upload Aug 7 22:44:47.819148413 (13h4m47.4084934s) [upload=13556]: Uploading segments... Aug 7 22:44:47.819394214 (13h4m47.408739201s) [upload=13556]: Got next segment. Inline: false Aug 7 22:44:47.819437876 (13h4m47.408782864s) [upload=13556]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:44:47.819442115 (13h4m47.408787092s) [upload=13556]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:44:48.269086796 (13h4m47.858431783s) [seg_pos={0 0}, upload=13556]: Begin upload segment... Aug 7 22:44:48.269139916 (13h4m47.858484903s) [seg_pos={0 0}, upload=13556]: Done begin upload segment. Aug 7 22:44:48.269169674 (13h4m47.858514661s) [noise=true , address=104.36.82.45:4938 , piece_id=NF4CEKTH5KGR2VLO64EISWJ64CRKFHKXBP3R77ZAZ3HVAFAYJJ4Q, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269176958 (13h4m47.858521965s) [noise=true , address=104.36.82.60:4343 , piece_id=7ZZ3C77QFADRAQU3ZQF3LRXC5S6NF3XBNFXZ6XICWDCI3HFZ6EUQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269619725 (13h4m47.858964712s) [noise=true , address=104.36.82.41:4768 , piece_id=GSW4YJR4SSW5ZKGPYI7L2KRRSHL3FTEFUE2RSARPDARD7CWOSXBA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269688626 (13h4m47.859033623s) [noise=true , address=104.36.82.9:4040 , piece_id=7YWI2TUXLX4QSI6GRFGIGLZRU3RUEUMJNJRVGAJON3NVRIZJ27DA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269932232 (13h4m47.85927722s) [noise=true , address=193.149.164.143:4696 , piece_id=D74PAAUDHRBQJSBQ7T5R6QDKUDUS4R7425SO7H53NHS4QZPTGBMQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.270291710 (13h4m47.859636717s) [noise=true , address=104.36.82.2:4742 , piece_id=TREABDVVCEXW3MN7U6TE5CWVGQXWW2Z5REAV7PEZWYRYNBK27ECQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271203444 (13h4m47.860548432s) [noise=true , address=104.36.82.44:4893 , piece_id=IRD65MAG7YYBVEOPHERHYVPFA65YC7JQPJB6XKUQLHGKKMYTFLFQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271407435 (13h4m47.860752422s) [noise=true , address=104.36.82.53:4122 , piece_id=DE5JKFJWUT4MIKUUQHWMXBTSK2MKDSTJ5VGJ5S2ALJXUTEJ3LUMA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271531252 (13h4m47.860876239s) [noise=true , address=104.36.82.59:4326 , piece_id=LJ5XKKU2EVRDGH5E4YAE5BRF5EIFM2BHG7BYMHOCQXIWUNS7DXRA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271734040 (13h4m47.861079027s) [noise=true , address=79.127.201.214:4295 , piece_id=6MJQFTPQRQL3OBVOWJ36WYWD3PVJBBCCMISXCJKC636VZPG474PA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271963008 (13h4m47.861307995s) [noise=true , address=104.36.82.8:4989 , piece_id=LPXKK2B443AVS624U7MPUP4GUEMUFJYVUBZY5ZJGZPV32WYW6T6A, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272159905 (13h4m47.861504882s) [noise=true , address=104.36.82.43:4848 , piece_id=KQKHRNP6BMU7GRFQXGW7U5KSNF53NGMJIRFNNE5M3Q6MHGQJUC4Q, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272343486 (13h4m47.861688463s) [noise=true , address=104.36.82.38:4715 , piece_id=BJDDYECUSWJH675QPKG4IP5PSTOS4Y77OMKQWG2BGZLAQE77GKHQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272525674 (13h4m47.861870661s) [noise=true , address=104.36.82.15:4277 , piece_id=NCLDOZRBEK4OF7C4Q5DY4RX3ND73SMUM6QIRL5FHVOFBQYKOKIOQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272744403 (13h4m47.86208939s) [noise=true , address=79.127.205.234:4597 , piece_id=CI2MXO5CGMRNZRAPXUCIGASHDL4KCMGAWVNMJESCLF3HTBRHWAGQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272934386 (13h4m47.862279373s) [noise=true , address=104.36.82.46:4483 , piece_id=ZDGTDBBYXOWDCNUHAA7BLEGD4RAZOR6QLETRO4MG3MAGDE6RVLQQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.273120443 (13h4m47.86246543s) [noise=true , address=104.36.82.44:4901 , piece_id=UP7742ZXVVCLWBHJPYVKZH2C3W4M5NWE5QHEMSZGRWJM4Q72FIDA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.273299345 (13h4m47.862644322s) [noise=true , address=104.36.82.22:4512 , piece_id=3BXMDY3JO7ZVVX6GF3LUH5RTR2GHVF5VJ2U5XEUF62VIUIXRVWSQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:49.376334243 (13h4m48.96567923s) [upload=13556]: Next returned nil segment Aug 7 22:44:49.376348381 (13h4m48.965693368s) [upload=13556]: Waiting for error group managing segments... Aug 7 22:44:49.668631065 (13h4m49.257976052s) [noise=true , address=79.127.205.234:4597 , piece_id=CI2MXO5CGMRNZRAPXUCIGASHDL4KCMGAWVNMJESCLF3HTBRHWAGQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:49.675850678 (13h4m49.265195665s) [noise=true , address=193.149.164.143:4696 , piece_id=D74PAAUDHRBQJSBQ7T5R6QDKUDUS4R7425SO7H53NHS4QZPTGBMQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.149126209 (13h4m49.738471196s) [noise=true , address=79.127.201.214:4295 , piece_id=6MJQFTPQRQL3OBVOWJ36WYWD3PVJBBCCMISXCJKC636VZPG474PA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.188374551 (13h4m49.777719528s) [noise=true , address=104.36.82.43:4848 , piece_id=KQKHRNP6BMU7GRFQXGW7U5KSNF53NGMJIRFNNE5M3Q6MHGQJUC4Q, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.192657957 (13h4m49.782002944s) [noise=true , address=104.36.82.22:4512 , piece_id=3BXMDY3JO7ZVVX6GF3LUH5RTR2GHVF5VJ2U5XEUF62VIUIXRVWSQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.260069984 (13h4m49.849414972s) [noise=true , address=104.36.82.38:4715 , piece_id=BJDDYECUSWJH675QPKG4IP5PSTOS4Y77OMKQWG2BGZLAQE77GKHQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.295276123 (13h4m49.88462111s) [noise=true , address=104.36.82.45:4938 , piece_id=NF4CEKTH5KGR2VLO64EISWJ64CRKFHKXBP3R77ZAZ3HVAFAYJJ4Q, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.314186263 (13h4m49.90353125s) [noise=true , address=104.36.82.59:4326 , piece_id=LJ5XKKU2EVRDGH5E4YAE5BRF5EIFM2BHG7BYMHOCQXIWUNS7DXRA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.335401744 (13h4m49.924746731s) [noise=true , address=104.36.82.8:4989 , piece_id=LPXKK2B443AVS624U7MPUP4GUEMUFJYVUBZY5ZJGZPV32WYW6T6A, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.350133970 (13h4m49.939478957s) [noise=true , address=104.36.82.41:4768 , piece_id=GSW4YJR4SSW5ZKGPYI7L2KRRSHL3FTEFUE2RSARPDARD7CWOSXBA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.370519655 (13h4m49.959864642s) [noise=true , address=104.36.82.53:4122 , piece_id=DE5JKFJWUT4MIKUUQHWMXBTSK2MKDSTJ5VGJ5S2ALJXUTEJ3LUMA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.407688108 (13h4m49.997033095s) [noise=true , address=104.36.82.9:4040 , piece_id=7YWI2TUXLX4QSI6GRFGIGLZRU3RUEUMJNJRVGAJON3NVRIZJ27DA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.484565775 (13h4m50.073910762s) [noise=true , address=104.36.82.15:4277 , piece_id=NCLDOZRBEK4OF7C4Q5DY4RX3ND73SMUM6QIRL5FHVOFBQYKOKIOQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.666687912 (13h4m50.256032899s) [noise=true , address=104.36.82.60:4343 , piece_id=7ZZ3C77QFADRAQU3ZQF3LRXC5S6NF3XBNFXZ6XICWDCI3HFZ6EUQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.666718511 (13h4m50.256063498s) [seg_pos={0 0}, upload=13556]: Segment reached optimal threshold of 14 pieces. Aug 7 22:44:50.667127924 (13h4m50.256472911s) [noise=true , address=104.36.82.2:4742 , piece_id=TREABDVVCEXW3MN7U6TE5CWVGQXWW2Z5REAV7PEZWYRYNBK27ECQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsGo4te3PENFo9YRioYFrhkeRQvztnLVZJV4i38TV3TjKqBFT): protocol: expected piece hash; context canceled; EOF Aug 7 22:44:50.667260107 (13h4m50.256605094s) [noise=true , address=104.36.82.46:4483 , piece_id=ZDGTDBBYXOWDCNUHAA7BLEGD4RAZOR6QLETRO4MG3MAGDE6RVLQQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: write tcp 188.166.225.170:58454->104.36.82.46:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:44:50.667317968 (13h4m50.256662945s) [noise=true , address=104.36.82.44:4893 , piece_id=IRD65MAG7YYBVEOPHERHYVPFA65YC7JQPJB6XKUQLHGKKMYTFLFQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: rpc: dial tcp 104.36.82.44:4893: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:44:50.667373113 (13h4m50.256718091s) [noise=true , address=104.36.82.44:4901 , piece_id=UP7742ZXVVCLWBHJPYVKZH2C3W4M5NWE5QHEMSZGRWJM4Q72FIDA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: rpc: dial tcp 104.36.82.44:4901: operation was canceled; conn closed: op Aug 7 22:44:50.667382952 (13h4m50.256727939s) [seg_pos={0 0}, upload=13556]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:44:50.667435813 (13h4m50.2567808s) [upload=13556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:44:50.667439520 (13h4m50.256784507s) [upload=13556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:44:50.982639800 (13h4m50.571984787s) [upload=13556]: Done uploading segments... Aug 7 22:44:50.982656853 (13h4m50.572001841s) [upload=13556]: Upload finished. err: <nil> Duration: 7.246522369s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..X...................................] duration: 793.757369ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:48:22 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 793.775984ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXX.X..X..........................] duration: 1.944226903s FAIL (2025-08-07 22:48:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.944264234s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................XXX............................] duration: 5.770777228s FAIL (2025-08-07 22:48:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.770801954s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 231.738716ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:48:16 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 231.762112ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXX.XX..X.............................] duration: 1.862171427s FAIL (2025-08-07 22:48:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.862194832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [........................XXXXXXXXX..X.X......................] duration: 6.618638331s FAIL (2025-08-07 22:48:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.618659891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 17 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXX.X.............................] duration: 1.872879776s FAIL (2025-08-07 22:48:11 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.872980589s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXX.XXX.X.X.......................] duration: 2.333728151s FAIL (2025-08-07 22:48:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.333746997s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXX...........................] duration: 7.106618233s FAIL (2025-08-07 22:48:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.106640576s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 641.343139ms target: https://us1.storj.io/login FAIL (2025-08-07 22:48:03 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 641.365511ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [.........................XXXXXXXXX..X.X.....................] duration: 10.934883172s FAIL (2025-08-07 22:47:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 10.934907167s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X..............X.........X....X.................X....] duration: 8.411961664s FAIL (2025-08-07 22:47:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 81af0bd8b34b2e69389f7db10076d194a3d6fbdd58cd01527d1b66bf7a00fe45 != before 7800c7ceda312caa58fa18fda98736593dbd42db38936a6dbdeb420eade030ff ======LOGS====== pload=13556]: Done starting upload Aug 7 22:44:47.819148413 (13h4m47.4084934s) [upload=13556]: Uploading segments... Aug 7 22:44:47.819394214 (13h4m47.408739201s) [upload=13556]: Got next segment. Inline: false Aug 7 22:44:47.819437876 (13h4m47.408782864s) [upload=13556]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:44:47.819442115 (13h4m47.408787092s) [upload=13556]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:44:48.269086796 (13h4m47.858431783s) [seg_pos={0 0}, upload=13556]: Begin upload segment... Aug 7 22:44:48.269139916 (13h4m47.858484903s) [seg_pos={0 0}, upload=13556]: Done begin upload segment. Aug 7 22:44:48.269169674 (13h4m47.858514661s) [noise=true , address=104.36.82.45:4938 , piece_id=NF4CEKTH5KGR2VLO64EISWJ64CRKFHKXBP3R77ZAZ3HVAFAYJJ4Q, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269176958 (13h4m47.858521965s) [noise=true , address=104.36.82.60:4343 , piece_id=7ZZ3C77QFADRAQU3ZQF3LRXC5S6NF3XBNFXZ6XICWDCI3HFZ6EUQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269619725 (13h4m47.858964712s) [noise=true , address=104.36.82.41:4768 , piece_id=GSW4YJR4SSW5ZKGPYI7L2KRRSHL3FTEFUE2RSARPDARD7CWOSXBA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269688626 (13h4m47.859033623s) [noise=true , address=104.36.82.9:4040 , piece_id=7YWI2TUXLX4QSI6GRFGIGLZRU3RUEUMJNJRVGAJON3NVRIZJ27DA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269932232 (13h4m47.85927722s) [noise=true , address=193.149.164.143:4696 , piece_id=D74PAAUDHRBQJSBQ7T5R6QDKUDUS4R7425SO7H53NHS4QZPTGBMQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.270291710 (13h4m47.859636717s) [noise=true , address=104.36.82.2:4742 , piece_id=TREABDVVCEXW3MN7U6TE5CWVGQXWW2Z5REAV7PEZWYRYNBK27ECQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271203444 (13h4m47.860548432s) [noise=true , address=104.36.82.44:4893 , piece_id=IRD65MAG7YYBVEOPHERHYVPFA65YC7JQPJB6XKUQLHGKKMYTFLFQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271407435 (13h4m47.860752422s) [noise=true , address=104.36.82.53:4122 , piece_id=DE5JKFJWUT4MIKUUQHWMXBTSK2MKDSTJ5VGJ5S2ALJXUTEJ3LUMA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271531252 (13h4m47.860876239s) [noise=true , address=104.36.82.59:4326 , piece_id=LJ5XKKU2EVRDGH5E4YAE5BRF5EIFM2BHG7BYMHOCQXIWUNS7DXRA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271734040 (13h4m47.861079027s) [noise=true , address=79.127.201.214:4295 , piece_id=6MJQFTPQRQL3OBVOWJ36WYWD3PVJBBCCMISXCJKC636VZPG474PA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271963008 (13h4m47.861307995s) [noise=true , address=104.36.82.8:4989 , piece_id=LPXKK2B443AVS624U7MPUP4GUEMUFJYVUBZY5ZJGZPV32WYW6T6A, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272159905 (13h4m47.861504882s) [noise=true , address=104.36.82.43:4848 , piece_id=KQKHRNP6BMU7GRFQXGW7U5KSNF53NGMJIRFNNE5M3Q6MHGQJUC4Q, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272343486 (13h4m47.861688463s) [noise=true , address=104.36.82.38:4715 , piece_id=BJDDYECUSWJH675QPKG4IP5PSTOS4Y77OMKQWG2BGZLAQE77GKHQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272525674 (13h4m47.861870661s) [noise=true , address=104.36.82.15:4277 , piece_id=NCLDOZRBEK4OF7C4Q5DY4RX3ND73SMUM6QIRL5FHVOFBQYKOKIOQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272744403 (13h4m47.86208939s) [noise=true , address=79.127.205.234:4597 , piece_id=CI2MXO5CGMRNZRAPXUCIGASHDL4KCMGAWVNMJESCLF3HTBRHWAGQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272934386 (13h4m47.862279373s) [noise=true , address=104.36.82.46:4483 , piece_id=ZDGTDBBYXOWDCNUHAA7BLEGD4RAZOR6QLETRO4MG3MAGDE6RVLQQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.273120443 (13h4m47.86246543s) [noise=true , address=104.36.82.44:4901 , piece_id=UP7742ZXVVCLWBHJPYVKZH2C3W4M5NWE5QHEMSZGRWJM4Q72FIDA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.273299345 (13h4m47.862644322s) [noise=true , address=104.36.82.22:4512 , piece_id=3BXMDY3JO7ZVVX6GF3LUH5RTR2GHVF5VJ2U5XEUF62VIUIXRVWSQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:49.376334243 (13h4m48.96567923s) [upload=13556]: Next returned nil segment Aug 7 22:44:49.376348381 (13h4m48.965693368s) [upload=13556]: Waiting for error group managing segments... Aug 7 22:44:49.668631065 (13h4m49.257976052s) [noise=true , address=79.127.205.234:4597 , piece_id=CI2MXO5CGMRNZRAPXUCIGASHDL4KCMGAWVNMJESCLF3HTBRHWAGQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:49.675850678 (13h4m49.265195665s) [noise=true , address=193.149.164.143:4696 , piece_id=D74PAAUDHRBQJSBQ7T5R6QDKUDUS4R7425SO7H53NHS4QZPTGBMQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.149126209 (13h4m49.738471196s) [noise=true , address=79.127.201.214:4295 , piece_id=6MJQFTPQRQL3OBVOWJ36WYWD3PVJBBCCMISXCJKC636VZPG474PA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.188374551 (13h4m49.777719528s) [noise=true , address=104.36.82.43:4848 , piece_id=KQKHRNP6BMU7GRFQXGW7U5KSNF53NGMJIRFNNE5M3Q6MHGQJUC4Q, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.192657957 (13h4m49.782002944s) [noise=true , address=104.36.82.22:4512 , piece_id=3BXMDY3JO7ZVVX6GF3LUH5RTR2GHVF5VJ2U5XEUF62VIUIXRVWSQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.260069984 (13h4m49.849414972s) [noise=true , address=104.36.82.38:4715 , piece_id=BJDDYECUSWJH675QPKG4IP5PSTOS4Y77OMKQWG2BGZLAQE77GKHQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.295276123 (13h4m49.88462111s) [noise=true , address=104.36.82.45:4938 , piece_id=NF4CEKTH5KGR2VLO64EISWJ64CRKFHKXBP3R77ZAZ3HVAFAYJJ4Q, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.314186263 (13h4m49.90353125s) [noise=true , address=104.36.82.59:4326 , piece_id=LJ5XKKU2EVRDGH5E4YAE5BRF5EIFM2BHG7BYMHOCQXIWUNS7DXRA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.335401744 (13h4m49.924746731s) [noise=true , address=104.36.82.8:4989 , piece_id=LPXKK2B443AVS624U7MPUP4GUEMUFJYVUBZY5ZJGZPV32WYW6T6A, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.350133970 (13h4m49.939478957s) [noise=true , address=104.36.82.41:4768 , piece_id=GSW4YJR4SSW5ZKGPYI7L2KRRSHL3FTEFUE2RSARPDARD7CWOSXBA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.370519655 (13h4m49.959864642s) [noise=true , address=104.36.82.53:4122 , piece_id=DE5JKFJWUT4MIKUUQHWMXBTSK2MKDSTJ5VGJ5S2ALJXUTEJ3LUMA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.407688108 (13h4m49.997033095s) [noise=true , address=104.36.82.9:4040 , piece_id=7YWI2TUXLX4QSI6GRFGIGLZRU3RUEUMJNJRVGAJON3NVRIZJ27DA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.484565775 (13h4m50.073910762s) [noise=true , address=104.36.82.15:4277 , piece_id=NCLDOZRBEK4OF7C4Q5DY4RX3ND73SMUM6QIRL5FHVOFBQYKOKIOQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.666687912 (13h4m50.256032899s) [noise=true , address=104.36.82.60:4343 , piece_id=7ZZ3C77QFADRAQU3ZQF3LRXC5S6NF3XBNFXZ6XICWDCI3HFZ6EUQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.666718511 (13h4m50.256063498s) [seg_pos={0 0}, upload=13556]: Segment reached optimal threshold of 14 pieces. Aug 7 22:44:50.667127924 (13h4m50.256472911s) [noise=true , address=104.36.82.2:4742 , piece_id=TREABDVVCEXW3MN7U6TE5CWVGQXWW2Z5REAV7PEZWYRYNBK27ECQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsGo4te3PENFo9YRioYFrhkeRQvztnLVZJV4i38TV3TjKqBFT): protocol: expected piece hash; context canceled; EOF Aug 7 22:44:50.667260107 (13h4m50.256605094s) [noise=true , address=104.36.82.46:4483 , piece_id=ZDGTDBBYXOWDCNUHAA7BLEGD4RAZOR6QLETRO4MG3MAGDE6RVLQQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: write tcp 188.166.225.170:58454->104.36.82.46:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:44:50.667317968 (13h4m50.256662945s) [noise=true , address=104.36.82.44:4893 , piece_id=IRD65MAG7YYBVEOPHERHYVPFA65YC7JQPJB6XKUQLHGKKMYTFLFQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: rpc: dial tcp 104.36.82.44:4893: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:44:50.667373113 (13h4m50.256718091s) [noise=true , address=104.36.82.44:4901 , piece_id=UP7742ZXVVCLWBHJPYVKZH2C3W4M5NWE5QHEMSZGRWJM4Q72FIDA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: rpc: dial tcp 104.36.82.44:4901: operation was canceled; conn closed: op Aug 7 22:44:50.667382952 (13h4m50.256727939s) [seg_pos={0 0}, upload=13556]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:44:50.667435813 (13h4m50.2567808s) [upload=13556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:44:50.667439520 (13h4m50.256784507s) [upload=13556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:44:50.982639800 (13h4m50.571984787s) [upload=13556]: Done uploading segments... Aug 7 22:44:50.982656853 (13h4m50.572001841s) [upload=13556]: Upload finished. err: <nil> Duration: 8.411989937s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 251.696162ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:47:48 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 251.717432ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXXXXX...X......................] duration: 6.226024786s FAIL (2025-08-07 22:47:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.22604796s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXX.X..X.........................] duration: 2.037626238s FAIL (2025-08-07 22:47:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.037653501s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXX.X.......................] duration: 5.489312223s FAIL (2025-08-07 22:47:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.489334427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXX.XX..X............................] duration: 1.913678756s FAIL (2025-08-07 22:47:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.913710305s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................XXX...........................] duration: 5.591858218s FAIL (2025-08-07 22:47:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.591887613s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXX..........................] duration: 6.228761551s FAIL (2025-08-07 22:47:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.228786438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [..........................XXXXXXXXX..X.X....................] duration: 5.936678933s FAIL (2025-08-07 22:47:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.936704574s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXX...X.....................] duration: 6.167420471s FAIL (2025-08-07 22:47:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.16744028s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 17 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXX.X............................] duration: 1.872302702s FAIL (2025-08-07 22:47:16 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.872328892s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 654.589993ms target: https://us1.storj.io/login FAIL (2025-08-07 22:47:16 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 654.646412ms core satellite us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [......................X..X..................................] duration: 699.704769ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:47:11 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 699.727743ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXX.X..X........................] duration: 2.026058727s FAIL (2025-08-07 22:47:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.026089737s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXX.XXX.X.X......................] duration: 2.355916373s FAIL (2025-08-07 22:47:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.355941391s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X..............X.........X....X.................X...] duration: 9.811297508s FAIL (2025-08-07 22:46:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 81af0bd8b34b2e69389f7db10076d194a3d6fbdd58cd01527d1b66bf7a00fe45 != before 7800c7ceda312caa58fa18fda98736593dbd42db38936a6dbdeb420eade030ff ======LOGS====== pload=13556]: Done starting upload Aug 7 22:44:47.819148413 (13h4m47.4084934s) [upload=13556]: Uploading segments... Aug 7 22:44:47.819394214 (13h4m47.408739201s) [upload=13556]: Got next segment. Inline: false Aug 7 22:44:47.819437876 (13h4m47.408782864s) [upload=13556]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:44:47.819442115 (13h4m47.408787092s) [upload=13556]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:44:48.269086796 (13h4m47.858431783s) [seg_pos={0 0}, upload=13556]: Begin upload segment... Aug 7 22:44:48.269139916 (13h4m47.858484903s) [seg_pos={0 0}, upload=13556]: Done begin upload segment. Aug 7 22:44:48.269169674 (13h4m47.858514661s) [noise=true , address=104.36.82.45:4938 , piece_id=NF4CEKTH5KGR2VLO64EISWJ64CRKFHKXBP3R77ZAZ3HVAFAYJJ4Q, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269176958 (13h4m47.858521965s) [noise=true , address=104.36.82.60:4343 , piece_id=7ZZ3C77QFADRAQU3ZQF3LRXC5S6NF3XBNFXZ6XICWDCI3HFZ6EUQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269619725 (13h4m47.858964712s) [noise=true , address=104.36.82.41:4768 , piece_id=GSW4YJR4SSW5ZKGPYI7L2KRRSHL3FTEFUE2RSARPDARD7CWOSXBA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269688626 (13h4m47.859033623s) [noise=true , address=104.36.82.9:4040 , piece_id=7YWI2TUXLX4QSI6GRFGIGLZRU3RUEUMJNJRVGAJON3NVRIZJ27DA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269932232 (13h4m47.85927722s) [noise=true , address=193.149.164.143:4696 , piece_id=D74PAAUDHRBQJSBQ7T5R6QDKUDUS4R7425SO7H53NHS4QZPTGBMQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.270291710 (13h4m47.859636717s) [noise=true , address=104.36.82.2:4742 , piece_id=TREABDVVCEXW3MN7U6TE5CWVGQXWW2Z5REAV7PEZWYRYNBK27ECQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271203444 (13h4m47.860548432s) [noise=true , address=104.36.82.44:4893 , piece_id=IRD65MAG7YYBVEOPHERHYVPFA65YC7JQPJB6XKUQLHGKKMYTFLFQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271407435 (13h4m47.860752422s) [noise=true , address=104.36.82.53:4122 , piece_id=DE5JKFJWUT4MIKUUQHWMXBTSK2MKDSTJ5VGJ5S2ALJXUTEJ3LUMA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271531252 (13h4m47.860876239s) [noise=true , address=104.36.82.59:4326 , piece_id=LJ5XKKU2EVRDGH5E4YAE5BRF5EIFM2BHG7BYMHOCQXIWUNS7DXRA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271734040 (13h4m47.861079027s) [noise=true , address=79.127.201.214:4295 , piece_id=6MJQFTPQRQL3OBVOWJ36WYWD3PVJBBCCMISXCJKC636VZPG474PA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271963008 (13h4m47.861307995s) [noise=true , address=104.36.82.8:4989 , piece_id=LPXKK2B443AVS624U7MPUP4GUEMUFJYVUBZY5ZJGZPV32WYW6T6A, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272159905 (13h4m47.861504882s) [noise=true , address=104.36.82.43:4848 , piece_id=KQKHRNP6BMU7GRFQXGW7U5KSNF53NGMJIRFNNE5M3Q6MHGQJUC4Q, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272343486 (13h4m47.861688463s) [noise=true , address=104.36.82.38:4715 , piece_id=BJDDYECUSWJH675QPKG4IP5PSTOS4Y77OMKQWG2BGZLAQE77GKHQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272525674 (13h4m47.861870661s) [noise=true , address=104.36.82.15:4277 , piece_id=NCLDOZRBEK4OF7C4Q5DY4RX3ND73SMUM6QIRL5FHVOFBQYKOKIOQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272744403 (13h4m47.86208939s) [noise=true , address=79.127.205.234:4597 , piece_id=CI2MXO5CGMRNZRAPXUCIGASHDL4KCMGAWVNMJESCLF3HTBRHWAGQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272934386 (13h4m47.862279373s) [noise=true , address=104.36.82.46:4483 , piece_id=ZDGTDBBYXOWDCNUHAA7BLEGD4RAZOR6QLETRO4MG3MAGDE6RVLQQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.273120443 (13h4m47.86246543s) [noise=true , address=104.36.82.44:4901 , piece_id=UP7742ZXVVCLWBHJPYVKZH2C3W4M5NWE5QHEMSZGRWJM4Q72FIDA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.273299345 (13h4m47.862644322s) [noise=true , address=104.36.82.22:4512 , piece_id=3BXMDY3JO7ZVVX6GF3LUH5RTR2GHVF5VJ2U5XEUF62VIUIXRVWSQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:49.376334243 (13h4m48.96567923s) [upload=13556]: Next returned nil segment Aug 7 22:44:49.376348381 (13h4m48.965693368s) [upload=13556]: Waiting for error group managing segments... Aug 7 22:44:49.668631065 (13h4m49.257976052s) [noise=true , address=79.127.205.234:4597 , piece_id=CI2MXO5CGMRNZRAPXUCIGASHDL4KCMGAWVNMJESCLF3HTBRHWAGQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:49.675850678 (13h4m49.265195665s) [noise=true , address=193.149.164.143:4696 , piece_id=D74PAAUDHRBQJSBQ7T5R6QDKUDUS4R7425SO7H53NHS4QZPTGBMQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.149126209 (13h4m49.738471196s) [noise=true , address=79.127.201.214:4295 , piece_id=6MJQFTPQRQL3OBVOWJ36WYWD3PVJBBCCMISXCJKC636VZPG474PA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.188374551 (13h4m49.777719528s) [noise=true , address=104.36.82.43:4848 , piece_id=KQKHRNP6BMU7GRFQXGW7U5KSNF53NGMJIRFNNE5M3Q6MHGQJUC4Q, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.192657957 (13h4m49.782002944s) [noise=true , address=104.36.82.22:4512 , piece_id=3BXMDY3JO7ZVVX6GF3LUH5RTR2GHVF5VJ2U5XEUF62VIUIXRVWSQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.260069984 (13h4m49.849414972s) [noise=true , address=104.36.82.38:4715 , piece_id=BJDDYECUSWJH675QPKG4IP5PSTOS4Y77OMKQWG2BGZLAQE77GKHQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.295276123 (13h4m49.88462111s) [noise=true , address=104.36.82.45:4938 , piece_id=NF4CEKTH5KGR2VLO64EISWJ64CRKFHKXBP3R77ZAZ3HVAFAYJJ4Q, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.314186263 (13h4m49.90353125s) [noise=true , address=104.36.82.59:4326 , piece_id=LJ5XKKU2EVRDGH5E4YAE5BRF5EIFM2BHG7BYMHOCQXIWUNS7DXRA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.335401744 (13h4m49.924746731s) [noise=true , address=104.36.82.8:4989 , piece_id=LPXKK2B443AVS624U7MPUP4GUEMUFJYVUBZY5ZJGZPV32WYW6T6A, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.350133970 (13h4m49.939478957s) [noise=true , address=104.36.82.41:4768 , piece_id=GSW4YJR4SSW5ZKGPYI7L2KRRSHL3FTEFUE2RSARPDARD7CWOSXBA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.370519655 (13h4m49.959864642s) [noise=true , address=104.36.82.53:4122 , piece_id=DE5JKFJWUT4MIKUUQHWMXBTSK2MKDSTJ5VGJ5S2ALJXUTEJ3LUMA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.407688108 (13h4m49.997033095s) [noise=true , address=104.36.82.9:4040 , piece_id=7YWI2TUXLX4QSI6GRFGIGLZRU3RUEUMJNJRVGAJON3NVRIZJ27DA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.484565775 (13h4m50.073910762s) [noise=true , address=104.36.82.15:4277 , piece_id=NCLDOZRBEK4OF7C4Q5DY4RX3ND73SMUM6QIRL5FHVOFBQYKOKIOQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.666687912 (13h4m50.256032899s) [noise=true , address=104.36.82.60:4343 , piece_id=7ZZ3C77QFADRAQU3ZQF3LRXC5S6NF3XBNFXZ6XICWDCI3HFZ6EUQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.666718511 (13h4m50.256063498s) [seg_pos={0 0}, upload=13556]: Segment reached optimal threshold of 14 pieces. Aug 7 22:44:50.667127924 (13h4m50.256472911s) [noise=true , address=104.36.82.2:4742 , piece_id=TREABDVVCEXW3MN7U6TE5CWVGQXWW2Z5REAV7PEZWYRYNBK27ECQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsGo4te3PENFo9YRioYFrhkeRQvztnLVZJV4i38TV3TjKqBFT): protocol: expected piece hash; context canceled; EOF Aug 7 22:44:50.667260107 (13h4m50.256605094s) [noise=true , address=104.36.82.46:4483 , piece_id=ZDGTDBBYXOWDCNUHAA7BLEGD4RAZOR6QLETRO4MG3MAGDE6RVLQQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: write tcp 188.166.225.170:58454->104.36.82.46:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:44:50.667317968 (13h4m50.256662945s) [noise=true , address=104.36.82.44:4893 , piece_id=IRD65MAG7YYBVEOPHERHYVPFA65YC7JQPJB6XKUQLHGKKMYTFLFQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: rpc: dial tcp 104.36.82.44:4893: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:44:50.667373113 (13h4m50.256718091s) [noise=true , address=104.36.82.44:4901 , piece_id=UP7742ZXVVCLWBHJPYVKZH2C3W4M5NWE5QHEMSZGRWJM4Q72FIDA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: rpc: dial tcp 104.36.82.44:4901: operation was canceled; conn closed: op Aug 7 22:44:50.667382952 (13h4m50.256727939s) [seg_pos={0 0}, upload=13556]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:44:50.667435813 (13h4m50.2567808s) [upload=13556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:44:50.667439520 (13h4m50.256784507s) [upload=13556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:44:50.982639800 (13h4m50.571984787s) [upload=13556]: Done uploading segments... Aug 7 22:44:50.982656853 (13h4m50.572001841s) [upload=13556]: Upload finished. err: <nil> Duration: 9.811326163s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXX.XXX.X.X.....................] duration: 2.278308016s FAIL (2025-08-07 22:46:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.278329448s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXX.........................] duration: 6.813384364s FAIL (2025-08-07 22:46:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.813405374s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXX.XX..X...........................] duration: 1.859573805s FAIL (2025-08-07 22:46:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.859600487s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 17 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXX.X...........................] duration: 1.882334794s FAIL (2025-08-07 22:46:41 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.882364742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..............................XXX..........................] duration: 5.651132749s FAIL (2025-08-07 22:46:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.651161393s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..X.................................] duration: 689.840437ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:46:39 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 689.855407ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 317.947097ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:46:38 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 317.965382ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXX...X....................] duration: 6.309809337s FAIL (2025-08-07 22:46:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.309829405s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXX.X......................] duration: 5.67211014s FAIL (2025-08-07 22:46:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.67212996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X..............X.........X....X.................X..] duration: 7.862603906s FAIL (2025-08-07 22:46:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 81af0bd8b34b2e69389f7db10076d194a3d6fbdd58cd01527d1b66bf7a00fe45 != before 7800c7ceda312caa58fa18fda98736593dbd42db38936a6dbdeb420eade030ff ======LOGS====== pload=13556]: Done starting upload Aug 7 22:44:47.819148413 (13h4m47.4084934s) [upload=13556]: Uploading segments... Aug 7 22:44:47.819394214 (13h4m47.408739201s) [upload=13556]: Got next segment. Inline: false Aug 7 22:44:47.819437876 (13h4m47.408782864s) [upload=13556]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:44:47.819442115 (13h4m47.408787092s) [upload=13556]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:44:48.269086796 (13h4m47.858431783s) [seg_pos={0 0}, upload=13556]: Begin upload segment... Aug 7 22:44:48.269139916 (13h4m47.858484903s) [seg_pos={0 0}, upload=13556]: Done begin upload segment. Aug 7 22:44:48.269169674 (13h4m47.858514661s) [noise=true , address=104.36.82.45:4938 , piece_id=NF4CEKTH5KGR2VLO64EISWJ64CRKFHKXBP3R77ZAZ3HVAFAYJJ4Q, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269176958 (13h4m47.858521965s) [noise=true , address=104.36.82.60:4343 , piece_id=7ZZ3C77QFADRAQU3ZQF3LRXC5S6NF3XBNFXZ6XICWDCI3HFZ6EUQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269619725 (13h4m47.858964712s) [noise=true , address=104.36.82.41:4768 , piece_id=GSW4YJR4SSW5ZKGPYI7L2KRRSHL3FTEFUE2RSARPDARD7CWOSXBA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269688626 (13h4m47.859033623s) [noise=true , address=104.36.82.9:4040 , piece_id=7YWI2TUXLX4QSI6GRFGIGLZRU3RUEUMJNJRVGAJON3NVRIZJ27DA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269932232 (13h4m47.85927722s) [noise=true , address=193.149.164.143:4696 , piece_id=D74PAAUDHRBQJSBQ7T5R6QDKUDUS4R7425SO7H53NHS4QZPTGBMQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.270291710 (13h4m47.859636717s) [noise=true , address=104.36.82.2:4742 , piece_id=TREABDVVCEXW3MN7U6TE5CWVGQXWW2Z5REAV7PEZWYRYNBK27ECQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271203444 (13h4m47.860548432s) [noise=true , address=104.36.82.44:4893 , piece_id=IRD65MAG7YYBVEOPHERHYVPFA65YC7JQPJB6XKUQLHGKKMYTFLFQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271407435 (13h4m47.860752422s) [noise=true , address=104.36.82.53:4122 , piece_id=DE5JKFJWUT4MIKUUQHWMXBTSK2MKDSTJ5VGJ5S2ALJXUTEJ3LUMA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271531252 (13h4m47.860876239s) [noise=true , address=104.36.82.59:4326 , piece_id=LJ5XKKU2EVRDGH5E4YAE5BRF5EIFM2BHG7BYMHOCQXIWUNS7DXRA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271734040 (13h4m47.861079027s) [noise=true , address=79.127.201.214:4295 , piece_id=6MJQFTPQRQL3OBVOWJ36WYWD3PVJBBCCMISXCJKC636VZPG474PA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271963008 (13h4m47.861307995s) [noise=true , address=104.36.82.8:4989 , piece_id=LPXKK2B443AVS624U7MPUP4GUEMUFJYVUBZY5ZJGZPV32WYW6T6A, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272159905 (13h4m47.861504882s) [noise=true , address=104.36.82.43:4848 , piece_id=KQKHRNP6BMU7GRFQXGW7U5KSNF53NGMJIRFNNE5M3Q6MHGQJUC4Q, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272343486 (13h4m47.861688463s) [noise=true , address=104.36.82.38:4715 , piece_id=BJDDYECUSWJH675QPKG4IP5PSTOS4Y77OMKQWG2BGZLAQE77GKHQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272525674 (13h4m47.861870661s) [noise=true , address=104.36.82.15:4277 , piece_id=NCLDOZRBEK4OF7C4Q5DY4RX3ND73SMUM6QIRL5FHVOFBQYKOKIOQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272744403 (13h4m47.86208939s) [noise=true , address=79.127.205.234:4597 , piece_id=CI2MXO5CGMRNZRAPXUCIGASHDL4KCMGAWVNMJESCLF3HTBRHWAGQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272934386 (13h4m47.862279373s) [noise=true , address=104.36.82.46:4483 , piece_id=ZDGTDBBYXOWDCNUHAA7BLEGD4RAZOR6QLETRO4MG3MAGDE6RVLQQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.273120443 (13h4m47.86246543s) [noise=true , address=104.36.82.44:4901 , piece_id=UP7742ZXVVCLWBHJPYVKZH2C3W4M5NWE5QHEMSZGRWJM4Q72FIDA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.273299345 (13h4m47.862644322s) [noise=true , address=104.36.82.22:4512 , piece_id=3BXMDY3JO7ZVVX6GF3LUH5RTR2GHVF5VJ2U5XEUF62VIUIXRVWSQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:49.376334243 (13h4m48.96567923s) [upload=13556]: Next returned nil segment Aug 7 22:44:49.376348381 (13h4m48.965693368s) [upload=13556]: Waiting for error group managing segments... Aug 7 22:44:49.668631065 (13h4m49.257976052s) [noise=true , address=79.127.205.234:4597 , piece_id=CI2MXO5CGMRNZRAPXUCIGASHDL4KCMGAWVNMJESCLF3HTBRHWAGQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:49.675850678 (13h4m49.265195665s) [noise=true , address=193.149.164.143:4696 , piece_id=D74PAAUDHRBQJSBQ7T5R6QDKUDUS4R7425SO7H53NHS4QZPTGBMQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.149126209 (13h4m49.738471196s) [noise=true , address=79.127.201.214:4295 , piece_id=6MJQFTPQRQL3OBVOWJ36WYWD3PVJBBCCMISXCJKC636VZPG474PA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.188374551 (13h4m49.777719528s) [noise=true , address=104.36.82.43:4848 , piece_id=KQKHRNP6BMU7GRFQXGW7U5KSNF53NGMJIRFNNE5M3Q6MHGQJUC4Q, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.192657957 (13h4m49.782002944s) [noise=true , address=104.36.82.22:4512 , piece_id=3BXMDY3JO7ZVVX6GF3LUH5RTR2GHVF5VJ2U5XEUF62VIUIXRVWSQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.260069984 (13h4m49.849414972s) [noise=true , address=104.36.82.38:4715 , piece_id=BJDDYECUSWJH675QPKG4IP5PSTOS4Y77OMKQWG2BGZLAQE77GKHQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.295276123 (13h4m49.88462111s) [noise=true , address=104.36.82.45:4938 , piece_id=NF4CEKTH5KGR2VLO64EISWJ64CRKFHKXBP3R77ZAZ3HVAFAYJJ4Q, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.314186263 (13h4m49.90353125s) [noise=true , address=104.36.82.59:4326 , piece_id=LJ5XKKU2EVRDGH5E4YAE5BRF5EIFM2BHG7BYMHOCQXIWUNS7DXRA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.335401744 (13h4m49.924746731s) [noise=true , address=104.36.82.8:4989 , piece_id=LPXKK2B443AVS624U7MPUP4GUEMUFJYVUBZY5ZJGZPV32WYW6T6A, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.350133970 (13h4m49.939478957s) [noise=true , address=104.36.82.41:4768 , piece_id=GSW4YJR4SSW5ZKGPYI7L2KRRSHL3FTEFUE2RSARPDARD7CWOSXBA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.370519655 (13h4m49.959864642s) [noise=true , address=104.36.82.53:4122 , piece_id=DE5JKFJWUT4MIKUUQHWMXBTSK2MKDSTJ5VGJ5S2ALJXUTEJ3LUMA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.407688108 (13h4m49.997033095s) [noise=true , address=104.36.82.9:4040 , piece_id=7YWI2TUXLX4QSI6GRFGIGLZRU3RUEUMJNJRVGAJON3NVRIZJ27DA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.484565775 (13h4m50.073910762s) [noise=true , address=104.36.82.15:4277 , piece_id=NCLDOZRBEK4OF7C4Q5DY4RX3ND73SMUM6QIRL5FHVOFBQYKOKIOQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.666687912 (13h4m50.256032899s) [noise=true , address=104.36.82.60:4343 , piece_id=7ZZ3C77QFADRAQU3ZQF3LRXC5S6NF3XBNFXZ6XICWDCI3HFZ6EUQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.666718511 (13h4m50.256063498s) [seg_pos={0 0}, upload=13556]: Segment reached optimal threshold of 14 pieces. Aug 7 22:44:50.667127924 (13h4m50.256472911s) [noise=true , address=104.36.82.2:4742 , piece_id=TREABDVVCEXW3MN7U6TE5CWVGQXWW2Z5REAV7PEZWYRYNBK27ECQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsGo4te3PENFo9YRioYFrhkeRQvztnLVZJV4i38TV3TjKqBFT): protocol: expected piece hash; context canceled; EOF Aug 7 22:44:50.667260107 (13h4m50.256605094s) [noise=true , address=104.36.82.46:4483 , piece_id=ZDGTDBBYXOWDCNUHAA7BLEGD4RAZOR6QLETRO4MG3MAGDE6RVLQQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: write tcp 188.166.225.170:58454->104.36.82.46:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:44:50.667317968 (13h4m50.256662945s) [noise=true , address=104.36.82.44:4893 , piece_id=IRD65MAG7YYBVEOPHERHYVPFA65YC7JQPJB6XKUQLHGKKMYTFLFQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: rpc: dial tcp 104.36.82.44:4893: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:44:50.667373113 (13h4m50.256718091s) [noise=true , address=104.36.82.44:4901 , piece_id=UP7742ZXVVCLWBHJPYVKZH2C3W4M5NWE5QHEMSZGRWJM4Q72FIDA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: rpc: dial tcp 104.36.82.44:4901: operation was canceled; conn closed: op Aug 7 22:44:50.667382952 (13h4m50.256727939s) [seg_pos={0 0}, upload=13556]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:44:50.667435813 (13h4m50.2567808s) [upload=13556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:44:50.667439520 (13h4m50.256784507s) [upload=13556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:44:50.982639800 (13h4m50.571984787s) [upload=13556]: Done uploading segments... Aug 7 22:44:50.982656853 (13h4m50.572001841s) [upload=13556]: Upload finished. err: <nil> Duration: 7.86263208s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXX.X..X.......................] duration: 1.971272057s FAIL (2025-08-07 22:46:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.971444095s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)