Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX..XX..X..............................XXX.....] duration: 5.655008178s FAIL (2025-08-07 22:32:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.65503505s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.864399531s FAIL (2025-08-07 22:32:47 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 1.864431964s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X................................XXXXXXXXXXXXXXXXXXX...X....] duration: 5.951671715s FAIL (2025-08-07 22:32:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.951690049s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXXX.....X................................XXXXXXXXX..X.X....] duration: 9.351457032s FAIL (2025-08-07 22:32:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 9.351481349s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 20 out of 60 is failed (threshold=3) [.XX..X...............................XXXXXXXXXXXXXXXX.X.....] duration: 1.874332386s FAIL (2025-08-07 22:32:42 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.874357161s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [XX...X..X............................XXXXXXXXXXXXXXXXX.X....] duration: 5.571077995s FAIL (2025-08-07 22:32:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.571098083s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X..X..............] duration: 716.144569ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:32:37 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 716.17196ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXX.X..X.......] duration: 2.011037068s FAIL (2025-08-07 22:32:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.011183608s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X.X................................XXXXXXXXXXXXXXXXX......] duration: 7.661150701s FAIL (2025-08-07 22:32:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.661175348s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X....................X..............X.........X....X..] duration: 8.212309655s FAIL (2025-08-07 22:32:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== ding piece... Aug 7 22:30:27.106182415 (12h50m26.695527402s) [noise=true , address=64.49.13.137:4494 , piece_id=LFDJNPET6CVU2GRXM2KJ3LGNJPTGJ5EM4PWPHF3XVYMPHS366DZA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.106447803 (12h50m26.69579278s) [noise=true , address=104.36.82.61:4404 , piece_id=ZLDF3FSQTOGUD4OMZYNQZZ6MUTVUF6NICTZBHRPBUM5OR3SDYVIA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.106779869 (12h50m26.696124856s) [noise=true , address=104.36.82.52:4114 , piece_id=C5IZX5N3O4JEE2RDMVYZGSOWXVKQSGBJD67NOUNFT4LHMRNZL37A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.107212025 (12h50m26.696557012s) [noise=true , address=104.36.82.9:4033 , piece_id=WDDQWPVVTQKSC4JUHUXZ27G6GJXGGBCXBBSQMVEX7ONSCLEFFTAA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.108229842 (12h50m26.697574829s) [noise=true , address=104.36.82.37:4684 , piece_id=3VZGRTMMP3FQSDOTGNTVLJ43IHMA5FSUBOZ4P2SKWCYG5COVF7LQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.108480002 (12h50m26.697824989s) [noise=true , address=104.36.82.44:4884 , piece_id=HRBXMCCRYU4FBS73Q2IB5BK5BUUHAALRNODQ5XERZICLQ6PUVOBQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109071673 (12h50m26.69841666s) [noise=true , address=45.91.116.13:4160 , piece_id=UGSRSWK3RVAIVHVFWBHMUGHTZLZNCJN3DSSJTBF3EP3DY4UHMG5A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109303346 (12h50m26.698648333s) [noise=true , address=104.36.82.46:4504 , piece_id=W7DK423UD6SXLWL3LVYWLGTCFNUDATLWKEMQIPYXX4WZ555GD34A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109532515 (12h50m26.698877492s) [noise=true , address=79.127.201.215:4356 , piece_id=FUQGELP6V2FJZ6PVPVKMYKWEAY25P7OAYA5ORTS26FL4FMGAF7XQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109694265 (12h50m26.699039252s) [noise=true , address=79.127.205.235:4659 , piece_id=PDPXLV3GK4XVZTDBS2GC564GUEVII3XLOI54ALEUL2YWMBDFUNNQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109942329 (12h50m26.699287316s) [noise=true , address=79.127.205.234:4588 , piece_id=SCZQTO3IY6647Q6MRG3J2AMSWJAWIYKPTX2S2YQHIENXEFZ3C53A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110167320 (12h50m26.699512307s) [noise=true , address=104.36.82.34:4576 , piece_id=G3A3X2KHUXIYXEF4FODXHPISXNSWNTLTP5RIXXPIWGMDWWEYBGUQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110377551 (12h50m26.699722538s) [noise=true , address=104.36.82.16:4325 , piece_id=M4UMVR3RXMLSL3BI55AHRHPL46AAO6MU7V3V3GBZ7HQCR7WRZWSQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110578186 (12h50m26.699923173s) [noise=true , address=104.36.82.35:4583 , piece_id=L4R2PO7KNIIOIN7BDCYLRCO2MRR7XUERKE4JA5LZ4AW6K37ZKBZQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:28.206727754 (12h50m27.796072741s) [upload=13303]: Next returned nil segment Aug 7 22:30:28.206744677 (12h50m27.796089664s) [upload=13303]: Waiting for error group managing segments... Aug 7 22:30:28.490025109 (12h50m28.079370096s) [noise=true , address=79.127.205.235:4659 , piece_id=PDPXLV3GK4XVZTDBS2GC564GUEVII3XLOI54ALEUL2YWMBDFUNNQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.506048426 (12h50m28.095393403s) [noise=true , address=79.127.205.234:4588 , piece_id=SCZQTO3IY6647Q6MRG3J2AMSWJAWIYKPTX2S2YQHIENXEFZ3C53A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.547644951 (12h50m28.136989938s) [noise=true , address=64.49.13.169:4709 , piece_id=AWEF3LXORD7TDQACIKTDS4NM5XHF676LYFTKKULKNCQ6HA6ABNLQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.617351799 (12h50m28.206696786s) [noise=true , address=64.49.13.137:4494 , piece_id=LFDJNPET6CVU2GRXM2KJ3LGNJPTGJ5EM4PWPHF3XVYMPHS366DZA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.022989834 (12h50m28.612334841s) [noise=true , address=79.127.201.215:4356 , piece_id=FUQGELP6V2FJZ6PVPVKMYKWEAY25P7OAYA5ORTS26FL4FMGAF7XQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.123377683 (12h50m28.71272267s) [noise=true , address=104.36.82.62:4435 , piece_id=A6FYWW6YDPCOCWN3NYSVWHP6G3HW64V5PRS5PEMBFPVHILQOJSYA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.128754752 (12h50m28.718099729s) [noise=true , address=104.36.82.58:4266 , piece_id=IHNIYP6QQ5HRNGMAHY3NNXQMJM6TOGP4JM5INFUQONEKONUD3BEA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.473373880 (12h50m29.062718857s) [noise=true , address=104.36.82.52:4114 , piece_id=C5IZX5N3O4JEE2RDMVYZGSOWXVKQSGBJD67NOUNFT4LHMRNZL37A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.539252935 (12h50m29.128597922s) [noise=true , address=104.36.82.34:4576 , piece_id=G3A3X2KHUXIYXEF4FODXHPISXNSWNTLTP5RIXXPIWGMDWWEYBGUQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.597507541 (12h50m29.186852528s) [noise=true , address=104.36.82.35:4583 , piece_id=L4R2PO7KNIIOIN7BDCYLRCO2MRR7XUERKE4JA5LZ4AW6K37ZKBZQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.641259711 (12h50m29.230604718s) [noise=true , address=45.91.116.13:4160 , piece_id=UGSRSWK3RVAIVHVFWBHMUGHTZLZNCJN3DSSJTBF3EP3DY4UHMG5A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:30.117032753 (12h50m29.70637773s) [noise=true , address=104.36.82.46:4504 , piece_id=W7DK423UD6SXLWL3LVYWLGTCFNUDATLWKEMQIPYXX4WZ555GD34A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:30.995539243 (12h50m30.58488423s) [noise=true , address=104.36.82.37:4684 , piece_id=3VZGRTMMP3FQSDOTGNTVLJ43IHMA5FSUBOZ4P2SKWCYG5COVF7LQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:31.436214934 (12h50m31.025559921s) [noise=true , address=104.36.82.61:4404 , piece_id=ZLDF3FSQTOGUD4OMZYNQZZ6MUTVUF6NICTZBHRPBUM5OR3SDYVIA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:31.436243469 (12h50m31.025588456s) [seg_pos={0 0}, upload=13303]: Segment reached optimal threshold of 14 pieces. Aug 7 22:30:31.436426830 (12h50m31.025771817s) [noise=true , address=104.36.82.16:4325 , piece_id=M4UMVR3RXMLSL3BI55AHRHPL46AAO6MU7V3V3GBZ7HQCR7WRZWSQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aTYNxPbXJGM29Xqovc5VK4cun64wr2DoduaVCvKDj38xCLGKJ): protocol: expected piece hash; context canceled; EOF Aug 7 22:30:31.436556607 (12h50m31.025901594s) [noise=true , address=104.36.82.9:4033 , piece_id=WDDQWPVVTQKSC4JUHUXZ27G6GJXGGBCXBBSQMVEX7ONSCLEFFTAA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C): protocol: write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection; write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection Aug 7 22:30:31.436606383 (12h50m31.02595137s) [noise=true , address=104.36.82.44:4884 , piece_id=HRBXMCCRYU4FBS73Q2IB5BK5BUUHAALRNODQ5XERZICLQ6PUVOBQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG): protocol: write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection; write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection Aug 7 22:30:31.436613496 (12h50m31.025958483s) [noise=true , address=104.36.82.5:4853 , piece_id=FZOYMSE2HPKJUXG7RXFQH77H7GTC34UFQVKAXJBKQ4YYI7ICAKMA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection; write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection Aug 7 22:30:31.436624858 (12h50m31.025969845s) [seg_pos={0 0}, upload=13303]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:30:31.436720120 (12h50m31.026065107s) [upload=13303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:30:31.436724318 (12h50m31.026069295s) [upload=13303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:31.748725210 (12h50m31.338070187s) [upload=13303]: Done uploading segments... Aug 7 22:30:31.748742002 (12h50m31.338086989s) [upload=13303]: Upload finished. err: <nil> Aug 7 22:31:18.785686607 (12h51m18.375031594s) [upload=13318]: Starting upload Aug 7 22:31:18.785729449 (12h51m18.375074437s) [upload=13318]: Done starting upload Aug 7 22:31:18.785814212 (12h51m18.375159189s) [upload=13318]: Uploading segments... Aug 7 22:31:18.785960702 (12h51m18.375305679s) [upload=13318]: Got next segment. Inline: false Aug 7 22:31:18.785988354 (12h51m18.375333341s) [upload=13318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:18.785993003 (12h51m18.37533798s) [upload=13318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:31:19.001950809 (12h51m18.591295786s) [upload=13318]: Next segment err: metaclient: Too Many Requests Aug 7 22:31:19.001977260 (12h51m18.591322247s) [upload=13318]: Done uploading segments... Aug 7 22:31:19.001992078 (12h51m18.591337055s) [upload=13318]: Upload finished. err: metaclient: Too Many Requests Duration: 8.212332899s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXX.XX..X......] duration: 1.878357875s FAIL (2025-08-07 22:32:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.878387211s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X................................XXXXXXXXXXXXXXXXXXX...X...] duration: 6.128304048s FAIL (2025-08-07 22:32:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.128326291s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 230.184175ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:32:19 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 230.208481ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X..X.............] duration: 731.484028ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:32:16 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 731.518295ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXX.X..X......] duration: 1.894927708s FAIL (2025-08-07 22:32:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.894954668s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [XXX...X..X............................XXXXXXXXXXXXXXXXX.X...] duration: 5.864207804s FAIL (2025-08-07 22:32:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.864226661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X....................X..............X.........X....X.] duration: 7.158004102s FAIL (2025-08-07 22:32:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== ding piece... Aug 7 22:30:27.106182415 (12h50m26.695527402s) [noise=true , address=64.49.13.137:4494 , piece_id=LFDJNPET6CVU2GRXM2KJ3LGNJPTGJ5EM4PWPHF3XVYMPHS366DZA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.106447803 (12h50m26.69579278s) [noise=true , address=104.36.82.61:4404 , piece_id=ZLDF3FSQTOGUD4OMZYNQZZ6MUTVUF6NICTZBHRPBUM5OR3SDYVIA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.106779869 (12h50m26.696124856s) [noise=true , address=104.36.82.52:4114 , piece_id=C5IZX5N3O4JEE2RDMVYZGSOWXVKQSGBJD67NOUNFT4LHMRNZL37A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.107212025 (12h50m26.696557012s) [noise=true , address=104.36.82.9:4033 , piece_id=WDDQWPVVTQKSC4JUHUXZ27G6GJXGGBCXBBSQMVEX7ONSCLEFFTAA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.108229842 (12h50m26.697574829s) [noise=true , address=104.36.82.37:4684 , piece_id=3VZGRTMMP3FQSDOTGNTVLJ43IHMA5FSUBOZ4P2SKWCYG5COVF7LQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.108480002 (12h50m26.697824989s) [noise=true , address=104.36.82.44:4884 , piece_id=HRBXMCCRYU4FBS73Q2IB5BK5BUUHAALRNODQ5XERZICLQ6PUVOBQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109071673 (12h50m26.69841666s) [noise=true , address=45.91.116.13:4160 , piece_id=UGSRSWK3RVAIVHVFWBHMUGHTZLZNCJN3DSSJTBF3EP3DY4UHMG5A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109303346 (12h50m26.698648333s) [noise=true , address=104.36.82.46:4504 , piece_id=W7DK423UD6SXLWL3LVYWLGTCFNUDATLWKEMQIPYXX4WZ555GD34A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109532515 (12h50m26.698877492s) [noise=true , address=79.127.201.215:4356 , piece_id=FUQGELP6V2FJZ6PVPVKMYKWEAY25P7OAYA5ORTS26FL4FMGAF7XQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109694265 (12h50m26.699039252s) [noise=true , address=79.127.205.235:4659 , piece_id=PDPXLV3GK4XVZTDBS2GC564GUEVII3XLOI54ALEUL2YWMBDFUNNQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109942329 (12h50m26.699287316s) [noise=true , address=79.127.205.234:4588 , piece_id=SCZQTO3IY6647Q6MRG3J2AMSWJAWIYKPTX2S2YQHIENXEFZ3C53A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110167320 (12h50m26.699512307s) [noise=true , address=104.36.82.34:4576 , piece_id=G3A3X2KHUXIYXEF4FODXHPISXNSWNTLTP5RIXXPIWGMDWWEYBGUQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110377551 (12h50m26.699722538s) [noise=true , address=104.36.82.16:4325 , piece_id=M4UMVR3RXMLSL3BI55AHRHPL46AAO6MU7V3V3GBZ7HQCR7WRZWSQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110578186 (12h50m26.699923173s) [noise=true , address=104.36.82.35:4583 , piece_id=L4R2PO7KNIIOIN7BDCYLRCO2MRR7XUERKE4JA5LZ4AW6K37ZKBZQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:28.206727754 (12h50m27.796072741s) [upload=13303]: Next returned nil segment Aug 7 22:30:28.206744677 (12h50m27.796089664s) [upload=13303]: Waiting for error group managing segments... Aug 7 22:30:28.490025109 (12h50m28.079370096s) [noise=true , address=79.127.205.235:4659 , piece_id=PDPXLV3GK4XVZTDBS2GC564GUEVII3XLOI54ALEUL2YWMBDFUNNQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.506048426 (12h50m28.095393403s) [noise=true , address=79.127.205.234:4588 , piece_id=SCZQTO3IY6647Q6MRG3J2AMSWJAWIYKPTX2S2YQHIENXEFZ3C53A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.547644951 (12h50m28.136989938s) [noise=true , address=64.49.13.169:4709 , piece_id=AWEF3LXORD7TDQACIKTDS4NM5XHF676LYFTKKULKNCQ6HA6ABNLQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.617351799 (12h50m28.206696786s) [noise=true , address=64.49.13.137:4494 , piece_id=LFDJNPET6CVU2GRXM2KJ3LGNJPTGJ5EM4PWPHF3XVYMPHS366DZA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.022989834 (12h50m28.612334841s) [noise=true , address=79.127.201.215:4356 , piece_id=FUQGELP6V2FJZ6PVPVKMYKWEAY25P7OAYA5ORTS26FL4FMGAF7XQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.123377683 (12h50m28.71272267s) [noise=true , address=104.36.82.62:4435 , piece_id=A6FYWW6YDPCOCWN3NYSVWHP6G3HW64V5PRS5PEMBFPVHILQOJSYA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.128754752 (12h50m28.718099729s) [noise=true , address=104.36.82.58:4266 , piece_id=IHNIYP6QQ5HRNGMAHY3NNXQMJM6TOGP4JM5INFUQONEKONUD3BEA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.473373880 (12h50m29.062718857s) [noise=true , address=104.36.82.52:4114 , piece_id=C5IZX5N3O4JEE2RDMVYZGSOWXVKQSGBJD67NOUNFT4LHMRNZL37A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.539252935 (12h50m29.128597922s) [noise=true , address=104.36.82.34:4576 , piece_id=G3A3X2KHUXIYXEF4FODXHPISXNSWNTLTP5RIXXPIWGMDWWEYBGUQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.597507541 (12h50m29.186852528s) [noise=true , address=104.36.82.35:4583 , piece_id=L4R2PO7KNIIOIN7BDCYLRCO2MRR7XUERKE4JA5LZ4AW6K37ZKBZQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.641259711 (12h50m29.230604718s) [noise=true , address=45.91.116.13:4160 , piece_id=UGSRSWK3RVAIVHVFWBHMUGHTZLZNCJN3DSSJTBF3EP3DY4UHMG5A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:30.117032753 (12h50m29.70637773s) [noise=true , address=104.36.82.46:4504 , piece_id=W7DK423UD6SXLWL3LVYWLGTCFNUDATLWKEMQIPYXX4WZ555GD34A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:30.995539243 (12h50m30.58488423s) [noise=true , address=104.36.82.37:4684 , piece_id=3VZGRTMMP3FQSDOTGNTVLJ43IHMA5FSUBOZ4P2SKWCYG5COVF7LQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:31.436214934 (12h50m31.025559921s) [noise=true , address=104.36.82.61:4404 , piece_id=ZLDF3FSQTOGUD4OMZYNQZZ6MUTVUF6NICTZBHRPBUM5OR3SDYVIA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:31.436243469 (12h50m31.025588456s) [seg_pos={0 0}, upload=13303]: Segment reached optimal threshold of 14 pieces. Aug 7 22:30:31.436426830 (12h50m31.025771817s) [noise=true , address=104.36.82.16:4325 , piece_id=M4UMVR3RXMLSL3BI55AHRHPL46AAO6MU7V3V3GBZ7HQCR7WRZWSQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aTYNxPbXJGM29Xqovc5VK4cun64wr2DoduaVCvKDj38xCLGKJ): protocol: expected piece hash; context canceled; EOF Aug 7 22:30:31.436556607 (12h50m31.025901594s) [noise=true , address=104.36.82.9:4033 , piece_id=WDDQWPVVTQKSC4JUHUXZ27G6GJXGGBCXBBSQMVEX7ONSCLEFFTAA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C): protocol: write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection; write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection Aug 7 22:30:31.436606383 (12h50m31.02595137s) [noise=true , address=104.36.82.44:4884 , piece_id=HRBXMCCRYU4FBS73Q2IB5BK5BUUHAALRNODQ5XERZICLQ6PUVOBQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG): protocol: write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection; write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection Aug 7 22:30:31.436613496 (12h50m31.025958483s) [noise=true , address=104.36.82.5:4853 , piece_id=FZOYMSE2HPKJUXG7RXFQH77H7GTC34UFQVKAXJBKQ4YYI7ICAKMA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection; write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection Aug 7 22:30:31.436624858 (12h50m31.025969845s) [seg_pos={0 0}, upload=13303]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:30:31.436720120 (12h50m31.026065107s) [upload=13303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:30:31.436724318 (12h50m31.026069295s) [upload=13303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:31.748725210 (12h50m31.338070187s) [upload=13303]: Done uploading segments... Aug 7 22:30:31.748742002 (12h50m31.338086989s) [upload=13303]: Upload finished. err: <nil> Aug 7 22:31:18.785686607 (12h51m18.375031594s) [upload=13318]: Starting upload Aug 7 22:31:18.785729449 (12h51m18.375074437s) [upload=13318]: Done starting upload Aug 7 22:31:18.785814212 (12h51m18.375159189s) [upload=13318]: Uploading segments... Aug 7 22:31:18.785960702 (12h51m18.375305679s) [upload=13318]: Got next segment. Inline: false Aug 7 22:31:18.785988354 (12h51m18.375333341s) [upload=13318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:18.785993003 (12h51m18.37533798s) [upload=13318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:31:19.001950809 (12h51m18.591295786s) [upload=13318]: Next segment err: metaclient: Too Many Requests Aug 7 22:31:19.001977260 (12h51m18.591322247s) [upload=13318]: Done uploading segments... Aug 7 22:31:19.001992078 (12h51m18.591337055s) [upload=13318]: Upload finished. err: metaclient: Too Many Requests Duration: 7.15802916s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [XXXX...X..X............................XXXXXXXXXXXXXXXXX.X..] duration: 26.369110447s FAIL (2025-08-07 22:32:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.369464684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 21 out of 60 is failed (threshold=3) [X.XX..X...............................XXXXXXXXXXXXXXXX.X....] duration: 1.948751076s FAIL (2025-08-07 22:31:59 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.948776246s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX..XX..X..............................XXX....] duration: 5.665124666s FAIL (2025-08-07 22:31:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.665148682s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXX.XXX.X.X.] duration: 2.479879522s FAIL (2025-08-07 22:31:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.479894029s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXX.XX..X.....] duration: 1.847090574s FAIL (2025-08-07 22:31:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.847117115s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXX.XXX.X.X] duration: 30.001055958s FAIL (2025-08-07 22:31:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001078501s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.73069737s FAIL (2025-08-07 22:31:36 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 2.730724272s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX..XX..X..............................XXX...] duration: 5.772337803s FAIL (2025-08-07 22:31:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.772426473s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXXXX.....X................................XXXXXXXXX..X.X...] duration: 6.025806849s FAIL (2025-08-07 22:31:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.025834322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X.X................................XXXXXXXXXXXXXXXXX.....] duration: 6.798294406s FAIL (2025-08-07 22:31:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.798323652s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X................................XXXXXXXXXXXXXXXXXXX...X..] duration: 6.088272071s FAIL (2025-08-07 22:31:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.088289754s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 302.715116ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:31:22 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 302.736397ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [............................................X..X............] duration: 719.162682ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:31:21 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 719.188741ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXX.XX..X....] duration: 1.881864828s FAIL (2025-08-07 22:31:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.881886137s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X....................X..............X.........X....X] duration: 868.403981ms FAIL (2025-08-07 22:31:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== ding piece... Aug 7 22:30:27.106182415 (12h50m26.695527402s) [noise=true , address=64.49.13.137:4494 , piece_id=LFDJNPET6CVU2GRXM2KJ3LGNJPTGJ5EM4PWPHF3XVYMPHS366DZA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.106447803 (12h50m26.69579278s) [noise=true , address=104.36.82.61:4404 , piece_id=ZLDF3FSQTOGUD4OMZYNQZZ6MUTVUF6NICTZBHRPBUM5OR3SDYVIA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.106779869 (12h50m26.696124856s) [noise=true , address=104.36.82.52:4114 , piece_id=C5IZX5N3O4JEE2RDMVYZGSOWXVKQSGBJD67NOUNFT4LHMRNZL37A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.107212025 (12h50m26.696557012s) [noise=true , address=104.36.82.9:4033 , piece_id=WDDQWPVVTQKSC4JUHUXZ27G6GJXGGBCXBBSQMVEX7ONSCLEFFTAA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.108229842 (12h50m26.697574829s) [noise=true , address=104.36.82.37:4684 , piece_id=3VZGRTMMP3FQSDOTGNTVLJ43IHMA5FSUBOZ4P2SKWCYG5COVF7LQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.108480002 (12h50m26.697824989s) [noise=true , address=104.36.82.44:4884 , piece_id=HRBXMCCRYU4FBS73Q2IB5BK5BUUHAALRNODQ5XERZICLQ6PUVOBQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109071673 (12h50m26.69841666s) [noise=true , address=45.91.116.13:4160 , piece_id=UGSRSWK3RVAIVHVFWBHMUGHTZLZNCJN3DSSJTBF3EP3DY4UHMG5A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109303346 (12h50m26.698648333s) [noise=true , address=104.36.82.46:4504 , piece_id=W7DK423UD6SXLWL3LVYWLGTCFNUDATLWKEMQIPYXX4WZ555GD34A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109532515 (12h50m26.698877492s) [noise=true , address=79.127.201.215:4356 , piece_id=FUQGELP6V2FJZ6PVPVKMYKWEAY25P7OAYA5ORTS26FL4FMGAF7XQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109694265 (12h50m26.699039252s) [noise=true , address=79.127.205.235:4659 , piece_id=PDPXLV3GK4XVZTDBS2GC564GUEVII3XLOI54ALEUL2YWMBDFUNNQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109942329 (12h50m26.699287316s) [noise=true , address=79.127.205.234:4588 , piece_id=SCZQTO3IY6647Q6MRG3J2AMSWJAWIYKPTX2S2YQHIENXEFZ3C53A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110167320 (12h50m26.699512307s) [noise=true , address=104.36.82.34:4576 , piece_id=G3A3X2KHUXIYXEF4FODXHPISXNSWNTLTP5RIXXPIWGMDWWEYBGUQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110377551 (12h50m26.699722538s) [noise=true , address=104.36.82.16:4325 , piece_id=M4UMVR3RXMLSL3BI55AHRHPL46AAO6MU7V3V3GBZ7HQCR7WRZWSQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110578186 (12h50m26.699923173s) [noise=true , address=104.36.82.35:4583 , piece_id=L4R2PO7KNIIOIN7BDCYLRCO2MRR7XUERKE4JA5LZ4AW6K37ZKBZQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:28.206727754 (12h50m27.796072741s) [upload=13303]: Next returned nil segment Aug 7 22:30:28.206744677 (12h50m27.796089664s) [upload=13303]: Waiting for error group managing segments... Aug 7 22:30:28.490025109 (12h50m28.079370096s) [noise=true , address=79.127.205.235:4659 , piece_id=PDPXLV3GK4XVZTDBS2GC564GUEVII3XLOI54ALEUL2YWMBDFUNNQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.506048426 (12h50m28.095393403s) [noise=true , address=79.127.205.234:4588 , piece_id=SCZQTO3IY6647Q6MRG3J2AMSWJAWIYKPTX2S2YQHIENXEFZ3C53A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.547644951 (12h50m28.136989938s) [noise=true , address=64.49.13.169:4709 , piece_id=AWEF3LXORD7TDQACIKTDS4NM5XHF676LYFTKKULKNCQ6HA6ABNLQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.617351799 (12h50m28.206696786s) [noise=true , address=64.49.13.137:4494 , piece_id=LFDJNPET6CVU2GRXM2KJ3LGNJPTGJ5EM4PWPHF3XVYMPHS366DZA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.022989834 (12h50m28.612334841s) [noise=true , address=79.127.201.215:4356 , piece_id=FUQGELP6V2FJZ6PVPVKMYKWEAY25P7OAYA5ORTS26FL4FMGAF7XQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.123377683 (12h50m28.71272267s) [noise=true , address=104.36.82.62:4435 , piece_id=A6FYWW6YDPCOCWN3NYSVWHP6G3HW64V5PRS5PEMBFPVHILQOJSYA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.128754752 (12h50m28.718099729s) [noise=true , address=104.36.82.58:4266 , piece_id=IHNIYP6QQ5HRNGMAHY3NNXQMJM6TOGP4JM5INFUQONEKONUD3BEA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.473373880 (12h50m29.062718857s) [noise=true , address=104.36.82.52:4114 , piece_id=C5IZX5N3O4JEE2RDMVYZGSOWXVKQSGBJD67NOUNFT4LHMRNZL37A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.539252935 (12h50m29.128597922s) [noise=true , address=104.36.82.34:4576 , piece_id=G3A3X2KHUXIYXEF4FODXHPISXNSWNTLTP5RIXXPIWGMDWWEYBGUQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.597507541 (12h50m29.186852528s) [noise=true , address=104.36.82.35:4583 , piece_id=L4R2PO7KNIIOIN7BDCYLRCO2MRR7XUERKE4JA5LZ4AW6K37ZKBZQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.641259711 (12h50m29.230604718s) [noise=true , address=45.91.116.13:4160 , piece_id=UGSRSWK3RVAIVHVFWBHMUGHTZLZNCJN3DSSJTBF3EP3DY4UHMG5A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:30.117032753 (12h50m29.70637773s) [noise=true , address=104.36.82.46:4504 , piece_id=W7DK423UD6SXLWL3LVYWLGTCFNUDATLWKEMQIPYXX4WZ555GD34A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:30.995539243 (12h50m30.58488423s) [noise=true , address=104.36.82.37:4684 , piece_id=3VZGRTMMP3FQSDOTGNTVLJ43IHMA5FSUBOZ4P2SKWCYG5COVF7LQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:31.436214934 (12h50m31.025559921s) [noise=true , address=104.36.82.61:4404 , piece_id=ZLDF3FSQTOGUD4OMZYNQZZ6MUTVUF6NICTZBHRPBUM5OR3SDYVIA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:31.436243469 (12h50m31.025588456s) [seg_pos={0 0}, upload=13303]: Segment reached optimal threshold of 14 pieces. Aug 7 22:30:31.436426830 (12h50m31.025771817s) [noise=true , address=104.36.82.16:4325 , piece_id=M4UMVR3RXMLSL3BI55AHRHPL46AAO6MU7V3V3GBZ7HQCR7WRZWSQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aTYNxPbXJGM29Xqovc5VK4cun64wr2DoduaVCvKDj38xCLGKJ): protocol: expected piece hash; context canceled; EOF Aug 7 22:30:31.436556607 (12h50m31.025901594s) [noise=true , address=104.36.82.9:4033 , piece_id=WDDQWPVVTQKSC4JUHUXZ27G6GJXGGBCXBBSQMVEX7ONSCLEFFTAA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C): protocol: write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection; write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection Aug 7 22:30:31.436606383 (12h50m31.02595137s) [noise=true , address=104.36.82.44:4884 , piece_id=HRBXMCCRYU4FBS73Q2IB5BK5BUUHAALRNODQ5XERZICLQ6PUVOBQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG): protocol: write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection; write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection Aug 7 22:30:31.436613496 (12h50m31.025958483s) [noise=true , address=104.36.82.5:4853 , piece_id=FZOYMSE2HPKJUXG7RXFQH77H7GTC34UFQVKAXJBKQ4YYI7ICAKMA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection; write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection Aug 7 22:30:31.436624858 (12h50m31.025969845s) [seg_pos={0 0}, upload=13303]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:30:31.436720120 (12h50m31.026065107s) [upload=13303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:30:31.436724318 (12h50m31.026069295s) [upload=13303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:31.748725210 (12h50m31.338070187s) [upload=13303]: Done uploading segments... Aug 7 22:30:31.748742002 (12h50m31.338086989s) [upload=13303]: Upload finished. err: <nil> Aug 7 22:31:18.785686607 (12h51m18.375031594s) [upload=13318]: Starting upload Aug 7 22:31:18.785729449 (12h51m18.375074437s) [upload=13318]: Done starting upload Aug 7 22:31:18.785814212 (12h51m18.375159189s) [upload=13318]: Uploading segments... Aug 7 22:31:18.785960702 (12h51m18.375305679s) [upload=13318]: Got next segment. Inline: false Aug 7 22:31:18.785988354 (12h51m18.375333341s) [upload=13318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:18.785993003 (12h51m18.37533798s) [upload=13318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:31:19.001950809 (12h51m18.591295786s) [upload=13318]: Next segment err: metaclient: Too Many Requests Aug 7 22:31:19.001977260 (12h51m18.591322247s) [upload=13318]: Done uploading segments... Aug 7 22:31:19.001992078 (12h51m18.591337055s) [upload=13318]: Upload finished. err: metaclient: Too Many Requests Duration: 868.420224ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXX.....X................................XXXXXXXXX..X.X..] duration: 7.160117179s FAIL (2025-08-07 22:31:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.160150224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 22 out of 60 is failed (threshold=3) [XX.XX..X...............................XXXXXXXXXXXXXXXX.X...] duration: 1.849083757s FAIL (2025-08-07 22:31:10 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.849114717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXX.X..X.....] duration: 1.895178198s FAIL (2025-08-07 22:31:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.895214557s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXX.XX..X...] duration: 2.047649858s FAIL (2025-08-07 22:31:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.047682111s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.642695112s FAIL (2025-08-07 22:30:48 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 2.642765777s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 20 out of 60 is failed (threshold=3) [X...................................XXXXXXXXXXXXXXXX.XX..X..] duration: 1.981417556s FAIL (2025-08-07 22:30:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.981446152s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [XXXXX...X..X............................XXXXXXXXXXXXXXXXX.X.] duration: 7.24805292s FAIL (2025-08-07 22:30:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.248063851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX..XX..X..............................XXX..] duration: 9.698203119s FAIL (2025-08-07 22:30:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 9.698227906s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 23 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXX.XXX.X.] duration: 21.93753127s FAIL (2025-08-07 22:30:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.937550687s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X....................X..............X.........X....] duration: 8.527536968s FAIL (2025-08-07 22:30:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after fae6c99054b879588f7c111c4a79418bf0fd4d28eb1851a73c67d30c14ea05af != before 176b372b3e5163a3234838015d6e51e5129f73237d3357c818a59792a9bdc2e7 ======LOGS====== 66674202 (12h47m8.25601919s) [upload=13254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:27:09.111381447 (12h47m8.700726464s) [seg_pos={0 0}, upload=13254]: Begin upload segment... Aug 7 22:27:09.111442474 (12h47m8.700787502s) [seg_pos={0 0}, upload=13254]: Done begin upload segment. Aug 7 22:27:09.111478343 (12h47m8.70082332s) [noise=true , address=45.91.116.8:4075 , piece_id=DJ2T2MHNQ5NB65L2XNFJQJWTYNISMKIXYJSVRVEHPHOUXZFIFIVQ, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.111489113 (12h47m8.70083409s) [noise=true , address=79.127.205.234:4589 , piece_id=4CZDHCEZFIL5XQH2GXZPW5ZG6TYDMPTUCOWMBB6TNSBSNQ3NZH5Q, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.111826419 (12h47m8.701171406s) [noise=true , address=104.36.82.16:4299 , piece_id=HWTRCPU2G5FE72LTIOWCNNMI7CRMNLHYUID6AUK6LJT2QSKOFIRQ, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.112022574 (12h47m8.701367561s) [noise=true , address=104.36.82.47:4520 , piece_id=JGTVVNRCTXE2BM635TT7XKVZGJUACCFSOZJZZBTHM6AE37X35BYQ, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.112131873 (12h47m8.70147685s) [noise=true , address=104.36.82.43:4853 , piece_id=BOTLE3NENHIOBEGJQP5Q24J2KP5UZT3XIZHIPVIYIATASEQXE7WA, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.112429743 (12h47m8.701774721s) [noise=true , address=45.91.116.12:4244 , piece_id=S2GKPDOZTOV5WYGB4GKC2LNHCGADU6E5SFK76QHPTL3U5PYQF6DQ, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.112451805 (12h47m8.701796792s) [noise=true , address=64.49.13.27:4964 , piece_id=LS6S6EAL6BEMKGRJQ7SSVNFXCWB3IYUDTJMJU4IEDTP2MYUPMVFA, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.112731170 (12h47m8.702076157s) [noise=true , address=104.36.82.47:4542 , piece_id=ZBRLWHVHWZAVDPKFNZMNAGWUMV7O4PIDBOEDA3TTOTQ2WG55F4LQ, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.112750306 (12h47m8.702095293s) [noise=true , address=104.36.82.56:4245 , piece_id=63ROHWEI4MNIBZ6KP2IJPOM4W5HDO5PXMGUZWW3IQDWBTT3KRSTQ, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.112986168 (12h47m8.702331155s) [noise=true , address=79.127.205.233:4630 , piece_id=FVCBU7ZNLCUKM3LAXRK3EV6XO3H72ESLW47K7M3YETOJU5YQRVVQ, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.113284328 (12h47m8.702629316s) [noise=true , address=104.36.82.36:4651 , piece_id=AY6PFZUSEDQF4HPHJUUUK2LPGPZZ7WUGTWBYID4SCYY2YGM5EU4A, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.113499539 (12h47m8.702844526s) [noise=true , address=104.36.82.46:4503 , piece_id=KSZUGWPYAXYTR5KBBYZSVFKXQY3QYDMNBCSGVU2F2AD53PF4ONSA, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.113693572 (12h47m8.703038559s) [noise=true , address=104.36.82.4:4817 , piece_id=NRBFHPDJPXREN3TOCL3I54QOJE7VTHHLZ5NJGMB7OSUL3UC6EKIA, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.113906488 (12h47m8.703251475s) [noise=true , address=104.36.82.23:4454 , piece_id=S5RYPQEKFDPMCNGMYO7ZK7HW5C7OPKBFPANQUEPKX24NR4OHQIEQ, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.114099548 (12h47m8.703444535s) [noise=true , address=104.36.82.44:4886 , piece_id=2TETA5G6CKOPUSRLB7HQ364DLJL3S2IKJXZ7WUEDWWOCSHD4OGPA, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.114405162 (12h47m8.703750149s) [noise=true , address=104.36.82.9:4038 , piece_id=IPRWTUYDC2GWKG6Z4X6RD2K7WRO6X75ZZ3GX4ACVHIWY7HT7H5VA, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.116342850 (12h47m8.705687827s) [noise=true , address=104.36.82.13:4189 , piece_id=VAPSZFGU7325GVG6LQHTQKOHB3CQ3WTBQOGR2YOKKXMJUEJGDXRQ, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:09.116618538 (12h47m8.705963525s) [noise=true , address=45.91.116.5:4326 , piece_id=5PU3LZYW5NSQMPBS4T3ZEFX2OB56TC3BFBGSLCUBISM7XBJ2WY2Q, seg_pos={0 0}, upload=13254]: Uploading piece... Aug 7 22:27:10.208962387 (12h47m9.798307374s) [upload=13254]: Next returned nil segment Aug 7 22:27:10.208979309 (12h47m9.798324296s) [upload=13254]: Waiting for error group managing segments... Aug 7 22:27:10.499059081 (12h47m10.08842689s) [noise=true , address=79.127.205.233:4630 , piece_id=FVCBU7ZNLCUKM3LAXRK3EV6XO3H72ESLW47K7M3YETOJU5YQRVVQ, seg_pos={0 0}, upload=13254]: Done uploading piece. err: <nil> Aug 7 22:27:10.545355538 (12h47m10.134700515s) [noise=true , address=79.127.205.234:4589 , piece_id=4CZDHCEZFIL5XQH2GXZPW5ZG6TYDMPTUCOWMBB6TNSBSNQ3NZH5Q, seg_pos={0 0}, upload=13254]: Done uploading piece. err: <nil> Aug 7 22:27:10.547522444 (12h47m10.136867431s) [noise=true , address=64.49.13.27:4964 , piece_id=LS6S6EAL6BEMKGRJQ7SSVNFXCWB3IYUDTJMJU4IEDTP2MYUPMVFA, seg_pos={0 0}, upload=13254]: Done uploading piece. err: <nil> Aug 7 22:27:11.054601573 (12h47m10.64394656s) [noise=true , address=104.36.82.36:4651 , piece_id=AY6PFZUSEDQF4HPHJUUUK2LPGPZZ7WUGTWBYID4SCYY2YGM5EU4A, seg_pos={0 0}, upload=13254]: Done uploading piece. err: <nil> Aug 7 22:27:11.232711104 (12h47m10.822056091s) [noise=true , address=45.91.116.12:4244 , piece_id=S2GKPDOZTOV5WYGB4GKC2LNHCGADU6E5SFK76QHPTL3U5PYQF6DQ, seg_pos={0 0}, upload=13254]: Done uploading piece. err: <nil> Aug 7 22:27:11.320637341 (12h47m10.909982328s) [noise=true , address=104.36.82.56:4245 , piece_id=63ROHWEI4MNIBZ6KP2IJPOM4W5HDO5PXMGUZWW3IQDWBTT3KRSTQ, seg_pos={0 0}, upload=13254]: Done uploading piece. err: <nil> Aug 7 22:27:11.440049689 (12h47m11.029394676s) [noise=true , address=45.91.116.5:4326 , piece_id=5PU3LZYW5NSQMPBS4T3ZEFX2OB56TC3BFBGSLCUBISM7XBJ2WY2Q, seg_pos={0 0}, upload=13254]: Done uploading piece. err: <nil> Aug 7 22:27:11.520717609 (12h47m11.110062586s) [noise=true , address=45.91.116.8:4075 , piece_id=DJ2T2MHNQ5NB65L2XNFJQJWTYNISMKIXYJSVRVEHPHOUXZFIFIVQ, seg_pos={0 0}, upload=13254]: Done uploading piece. err: <nil> Aug 7 22:27:11.551809857 (12h47m11.141154844s) [noise=true , address=104.36.82.44:4886 , piece_id=2TETA5G6CKOPUSRLB7HQ364DLJL3S2IKJXZ7WUEDWWOCSHD4OGPA, seg_pos={0 0}, upload=13254]: Done uploading piece. err: <nil> Aug 7 22:27:12.200726590 (12h47m11.790071577s) [noise=true , address=104.36.82.4:4817 , piece_id=NRBFHPDJPXREN3TOCL3I54QOJE7VTHHLZ5NJGMB7OSUL3UC6EKIA, seg_pos={0 0}, upload=13254]: Done uploading piece. err: <nil> Aug 7 22:27:12.942007911 (12h47m12.531352888s) [noise=true , address=104.36.82.43:4853 , piece_id=BOTLE3NENHIOBEGJQP5Q24J2KP5UZT3XIZHIPVIYIATASEQXE7WA, seg_pos={0 0}, upload=13254]: Done uploading piece. err: <nil> Aug 7 22:27:13.652159762 (12h47m13.241504739s) [noise=true , address=104.36.82.13:4189 , piece_id=VAPSZFGU7325GVG6LQHTQKOHB3CQ3WTBQOGR2YOKKXMJUEJGDXRQ, seg_pos={0 0}, upload=13254]: Done uploading piece. err: <nil> Aug 7 22:27:13.774803293 (12h47m13.36414828s) [noise=true , address=104.36.82.46:4503 , piece_id=KSZUGWPYAXYTR5KBBYZSVFKXQY3QYDMNBCSGVU2F2AD53PF4ONSA, seg_pos={0 0}, upload=13254]: Done uploading piece. err: <nil> Aug 7 22:27:15.074875582 (12h47m14.664220569s) [noise=true , address=104.36.82.16:4299 , piece_id=HWTRCPU2G5FE72LTIOWCNNMI7CRMNLHYUID6AUK6LJT2QSKOFIRQ, seg_pos={0 0}, upload=13254]: Done uploading piece. err: <nil> Aug 7 22:27:15.074899248 (12h47m14.664244235s) [seg_pos={0 0}, upload=13254]: Segment reached optimal threshold of 14 pieces. Aug 7 22:27:15.075292069 (12h47m14.664637056s) [noise=true , address=104.36.82.47:4542 , piece_id=ZBRLWHVHWZAVDPKFNZMNAGWUMV7O4PIDBOEDA3TTOTQ2WG55F4LQ, seg_pos={0 0}, upload=13254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FwaCA1FbzJbmRNzhQeVgygicJYHWkSS9fLnJbAG7KZPpokLU): protocol: rpc: dial tcp 104.36.82.47:4542: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:27:15.075311917 (12h47m14.664656894s) [noise=true , address=104.36.82.9:4038 , piece_id=IPRWTUYDC2GWKG6Z4X6RD2K7WRO6X75ZZ3GX4ACVHIWY7HT7H5VA, seg_pos={0 0}, upload=13254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTVM9YuevPA4vsWExjpyW5B9pqyEKKruUwnj1jNhE3j1ScCBg): protocol: read tcp 188.166.225.170:39178->104.36.82.9:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:27:15.075342195 (12h47m14.664687182s) [noise=true , address=104.36.82.47:4520 , piece_id=JGTVVNRCTXE2BM635TT7XKVZGJUACCFSOZJZZBTHM6AE37X35BYQ, seg_pos={0 0}, upload=13254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy): protocol: write tcp 188.166.225.170:38208->104.36.82.47:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:27:15.075369517 (12h47m14.664714504s) [noise=true , address=104.36.82.23:4454 , piece_id=S5RYPQEKFDPMCNGMYO7ZK7HW5C7OPKBFPANQUEPKX24NR4OHQIEQ, seg_pos={0 0}, upload=13254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UCkuYGdVYLoR8oc1jpCZCN2smNfVqFwC8a7q95fJb8haB3MY2): protocol: write tcp 188.166.225.170:56462->104.36.82.23:4454: use of closed network connection; write tcp 188.166.225.170:56462->104.36.82.23:4454: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56462->104.36.82.23:4454: use of closed network connection Aug 7 22:27:15.075385046 (12h47m14.664730033s) [seg_pos={0 0}, upload=13254]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:27:15.075440973 (12h47m14.66478596s) [upload=13254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:27:15.075444660 (12h47m14.664789647s) [upload=13254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:15.456096179 (12h47m15.045441166s) [upload=13254]: Done uploading segments... Aug 7 22:27:15.456114193 (12h47m15.04545917s) [upload=13254]: Upload finished. err: <nil> Duration:* 8.527563669s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXX...X..X............................XXXXXXXXXXXXXXXXX.X] duration: 1m0.001425206s FAIL (2025-08-07 22:30:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001435085s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X..X...........] duration: 752.726358ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:30:28 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 752.738832ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X................................XXXXXXXXXXXXXXXXXXX...X.] duration: 25.083221814s FAIL (2025-08-07 22:30:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.083264736s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.206658312s remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:30:27 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 1.206689161s edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXX.....X................................XXXXXXXXX..X.X.] duration: 6.807787277s FAIL (2025-08-07 22:30:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.807803319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXXXXXXX.....X................................XXXXXXXXX..X.X] duration: 4m25.729350871s FAIL (2025-08-07 22:30:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 4m25.729371531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X.X.X................................XXXXXXXXXXXXXXXXX....] duration: 6.846354819s FAIL (2025-08-07 22:30:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.846379747s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXX.X..X....] duration: 1.913127488s FAIL (2025-08-07 22:30:12 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.913152716s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)