Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 733812 reports (Page 13157 of 14677)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XX.....X.XXX............................................] duration: 5.829934813s FAIL (2025-08-04 16:50:36 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.829964078s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.022872519s FAIL (2025-08-04 16:50:28 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.022900362s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.............................................X......] duration: 5.757055561s FAIL (2025-08-04 16:50:24 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M Duration: 5.757081151s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.44116228s FAIL (2025-08-04 16:50:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.441184834s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.421044679s FAIL (2025-08-04 16:50:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.42109149s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.............................................X.....] duration: 5.803629485s FAIL (2025-08-04 16:50:08 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M Duration: 5.803660955s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........XX.X.XX.XX..X........................X.............] duration: 6.137741311s FAIL (2025-08-04 16:50:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.137832296s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.534499352s FAIL (2025-08-04 16:49:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.53451347s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.844717331s FAIL (2025-08-04 16:49:53 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.84474346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XX.....X.XXX...........................................] duration: 5.837372255s FAIL (2025-08-04 16:49:49 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.83740102s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........XX.X.XX.XX..X........................X............] duration: 5.916438664s FAIL (2025-08-04 16:49:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.916466919s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.697311131s FAIL (2025-08-04 16:49:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.697337772s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1.414704568s FAIL (2025-08-04 16:49:33 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.414733364s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.............................................X....] duration: 5.807632646s FAIL (2025-08-04 16:49:19 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M Duration: 5.807665309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.778352289s FAIL (2025-08-04 16:49:17 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.778381134s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......XX.....X.XXX..........................................] duration: 5.847371834s FAIL (2025-08-04 16:49:16 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.84740121s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.484005332s FAIL (2025-08-04 16:49:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.484034539s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.290993226s FAIL (2025-08-04 16:49:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.291015469s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.............................................X...] duration: 5.893755453s FAIL (2025-08-04 16:48:52 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M Duration: 5.893787665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.641900151s FAIL (2025-08-04 16:48:50 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.641919969s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........XX.X.XX.XX..X........................X...........] duration: 5.948901562s FAIL (2025-08-04 16:48:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.948922463s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.312508134s FAIL (2025-08-04 16:48:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.312538773s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.527579149s FAIL (2025-08-04 16:48:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.527599819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.............................................X..] duration: 5.624957525s FAIL (2025-08-04 16:48:25 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M Duration: 5.624984356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......XX.....X.XXX.........................................] duration: 5.882163191s FAIL (2025-08-04 16:48:23 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.882199159s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.............................................X.] duration: 5.918294694s FAIL (2025-08-04 16:48:10 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M Duration: 5.918320223s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.828184718s FAIL (2025-08-04 16:48:06 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.828212973s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.472423597s FAIL (2025-08-04 16:47:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.472446831s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [............XX.X.XX.XX..X........................X..........] duration: 5.975562902s FAIL (2025-08-04 16:47:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.975585015s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.920663667s FAIL (2025-08-04 16:47:24 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.920688393s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.777763992s FAIL (2025-08-04 16:47:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.7777783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........XX.....X.XXX........................................] duration: 5.765972753s FAIL (2025-08-04 16:47:20 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.765999624s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.............................................X] duration: 4.043565347s FAIL (2025-08-04 16:47:11 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M Duration: 4.043588721s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.284117291s FAIL (2025-08-04 16:47:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.284168429s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.632781773s FAIL (2025-08-04 16:46:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.632798405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.864557788s FAIL (2025-08-04 16:46:47 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.864584901s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.356529441s FAIL (2025-08-04 16:46:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.356551764s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............XX.X.XX.XX..X........................X.........] duration: 6.215010311s FAIL (2025-08-04 16:46:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.215036831s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.628663547s FAIL (2025-08-04 16:46:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.628716749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........XX.....X.XXX.......................................] duration: 5.770970318s FAIL (2025-08-04 16:46:19 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.771021396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.902394034s FAIL (2025-08-04 16:46:19 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.902424633s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.409276433s FAIL (2025-08-04 16:46:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.409295639s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..............XX.X.XX.XX..X........................X........] duration: 6.285572819s FAIL (2025-08-04 16:46:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.285593079s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.699361618s FAIL (2025-08-04 16:45:44 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.699390553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........XX.....X.XXX......................................] duration: 5.862203095s FAIL (2025-08-04 16:45:31 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.862226449s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.468026743s FAIL (2025-08-04 16:45:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.468050319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.584460362s FAIL (2025-08-04 16:45:08 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.584490249s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.412375484s FAIL (2025-08-04 16:45:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.412392997s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.462546757s FAIL (2025-08-04 16:45:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.462573939s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...............XX.X.XX.XX..X........................X.......] duration: 5.790720281s FAIL (2025-08-04 16:45:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.790740389s edge gateway us-select-1

1
...
13155
13156
13157
13158
13159
...
14677