Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXXX] duration: 30.230738746s FAIL (2025-07-27 00:30:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:29:14.180912281 (60h53m23.903475321s) [upload=58374]: Done uploading segments... Jul 27 00:29:14.180949542 (60h53m23.903512582s) [upload=58374]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:29:52.655472786 (60h54m2.378035826s) [upload=58394]: Starting upload Jul 27 00:29:52.655531158 (60h54m2.378094198s) [upload=58394]: Done starting upload Jul 27 00:29:52.655627351 (60h54m2.378190381s) [upload=58394]: Uploading segments... Jul 27 00:29:52.655706563 (60h54m2.378269603s) [upload=58394]: Got next segment. Inline: false Jul 27 00:29:52.655747512 (60h54m2.378310552s) [upload=58394]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:29:52.655752682 (60h54m2.378315722s) [upload=58394]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:29:52.888121222 (60h54m2.610684262s) [seg_pos={0 0}, upload=58394]: Begin upload segment... Jul 27 00:29:52.888162492 (60h54m2.610725532s) [seg_pos={0 0}, upload=58394]: Done begin upload segment. Jul 27 00:29:52.888185005 (60h54m2.610748045s) [noise=true , address=45.91.116.4:4940 , piece_id=ZX2KGDIA4KKRZB7VTMZMEANSMUQK47MNXS4XRX465UZNNMZFEJXQ, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.888194152 (60h54m2.610757192s) [noise=true , address=77.90.31.21:4879 , piece_id=VCXVA6S4AD6TZB64JMKAEKFW6AKIIQTCNPRHTU3AYNEHIQ4DHAVA, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.888250500 (60h54m2.61081353s) [noise=true , address=77.90.31.12:4589 , piece_id=4I7EGZ6EXMBIK6TTGBHGAI45IPRBTP67W6OT2YHN74RQFD2XPZEQ, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.888546647 (60h54m2.611109717s) [noise=true , address=77.90.31.14:4661 , piece_id=K3SYCFI2OP6C3Z7EFVKZJFDVZ3ZQWMCHFRRUHM4EP67BD6PJ2I4Q, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.888609988 (60h54m2.611173028s) [noise=true , address=104.36.82.14:4221 , piece_id=5XRWLPOJHXUWFNSHPXTOQBETN7AM24KNEONYBDUEXIQDQE2JCBMA, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.888820371 (60h54m2.611383411s) [noise=true , address=104.36.82.4:4801 , piece_id=ADCR6YLPHB7VXYCIXT6HLR3ACJ3FHBQLYKIPWOQWCBXPEOHQQBGA, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.888918027 (60h54m2.611481067s) [noise=true , address=45.91.116.6:4975 , piece_id=6FGXCSAM4SKKNPHFHF23LMAI4TDQV7N6MMZTX4PJGL5EXDFUSIDQ, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.889061982 (60h54m2.611625022s) [noise=true , address=104.36.82.16:4293 , piece_id=AO42DKLBS6QAXDMJ4K2TKFX5RE4UTWDWKYI4B6HKYCVVB3WWD4CA, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.889165400 (60h54m2.61172844s) [noise=true , address=45.91.116.7:4011 , piece_id=AWQG67T4DCFF53T4D47YJ7W3N367UXZUC27K4MGLK4HTJL6KA5OA, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.889307111 (60h54m2.611870151s) [noise=true , address=45.91.116.11:4117 , piece_id=2626XHGTTVACC5NOUZ7NYH6FSMUVJ4B32K5XSPHQSGBQ6CWGJ57A, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.889373619 (60h54m2.611936659s) [noise=true , address=77.90.31.11:4555 , piece_id=PXT2TZSUXDTKPIEWREAG224KZGJZ4333GG3PQ5V6PN3M4WTBMKBQ, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.889567510 (60h54m2.61213054s) [noise=true , address=77.90.31.19:4843 , piece_id=LKKIBE65SCED3XZ2LKAJBJEGQGDVHFWXDAJWNSF7TSWK65SPZF5Q, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.889786339 (60h54m2.612349379s) [noise=true , address=45.91.116.12:4226 , piece_id=7EB6ZQJLAHN5ZUY6V6XI2ZMSDP5A3YXJ5AWVLUC7KRA4H7EEV6FA, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.890008103 (60h54m2.612571143s) [noise=true , address=45.91.116.8:4047 , piece_id=JAQBSAZQVW7MPINI4BELMFMYBWYREGR3S7EARRDVLKO3FDOMWCZA, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.890207524 (60h54m2.612770554s) [noise=true , address=104.36.82.11:4115 , piece_id=ODD6JFRCK5XE3VA6N2HF6TDNIUPTZTTBDCEFWDT4UVWVN4NS5K6Q, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:52.890594855 (60h54m2.613157895s) [noise=true , address=77.90.31.15:4697 , piece_id=DT7CTZTYNNEJPKHK435O3XVYZNJ2TI55GJMLCM4UVXN5IBYBZ5FA, seg_pos={0 0}, upload=58394]: Uploading piece... Jul 27 00:29:54.628543392 (60h54m4.351106432s) [upload=58394]: Next returned nil segment Jul 27 00:29:54.628569472 (60h54m4.351132512s) [upload=58394]: Waiting for error group managing segments... Jul 27 00:29:54.909550974 (60h54m4.632114014s) [noise=true , address=45.91.116.6:4975 , piece_id=6FGXCSAM4SKKNPHFHF23LMAI4TDQV7N6MMZTX4PJGL5EXDFUSIDQ, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:54.915602013 (60h54m4.638165043s) [noise=true , address=45.91.116.8:4047 , piece_id=JAQBSAZQVW7MPINI4BELMFMYBWYREGR3S7EARRDVLKO3FDOMWCZA, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:54.995497366 (60h54m4.718060406s) [noise=true , address=45.91.116.11:4117 , piece_id=2626XHGTTVACC5NOUZ7NYH6FSMUVJ4B32K5XSPHQSGBQ6CWGJ57A, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:54.996183028 (60h54m4.718746068s) [noise=true , address=77.90.31.21:4879 , piece_id=VCXVA6S4AD6TZB64JMKAEKFW6AKIIQTCNPRHTU3AYNEHIQ4DHAVA, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:55.013792189 (60h54m4.736355229s) [noise=true , address=45.91.116.7:4011 , piece_id=AWQG67T4DCFF53T4D47YJ7W3N367UXZUC27K4MGLK4HTJL6KA5OA, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:55.053289159 (60h54m4.775852189s) [noise=true , address=77.90.31.14:4661 , piece_id=K3SYCFI2OP6C3Z7EFVKZJFDVZ3ZQWMCHFRRUHM4EP67BD6PJ2I4Q, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:55.066042380 (60h54m4.78860542s) [noise=true , address=45.91.116.4:4940 , piece_id=ZX2KGDIA4KKRZB7VTMZMEANSMUQK47MNXS4XRX465UZNNMZFEJXQ, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:55.066112193 (60h54m4.788675223s) [noise=true , address=45.91.116.12:4226 , piece_id=7EB6ZQJLAHN5ZUY6V6XI2ZMSDP5A3YXJ5AWVLUC7KRA4H7EEV6FA, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:56.415866718 (60h54m6.138429759s) [noise=true , address=77.90.31.12:4589 , piece_id=4I7EGZ6EXMBIK6TTGBHGAI45IPRBTP67W6OT2YHN74RQFD2XPZEQ, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:29:56.421881577 (60h54m6.144444677s) [noise=true , address=77.90.31.19:4843 , piece_id=LKKIBE65SCED3XZ2LKAJBJEGQGDVHFWXDAJWNSF7TSWK65SPZF5Q, seg_pos={0 0}, upload=58394]: Done uploading piece. err: <nil> Jul 27 00:30:22.013004291 (60h54m31.735567331s) [noise=true , address=104.36.82.14:4221 , piece_id=5XRWLPOJHXUWFNSHPXTOQBETN7AM24KNEONYBDUEXIQDQE2JCBMA, seg_pos={0 0}, upload=58394]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:22.013049246 (60h54m31.735612286s) [noise=true , address=104.36.82.11:4115 , piece_id=ODD6JFRCK5XE3VA6N2HF6TDNIUPTZTTBDCEFWDT4UVWVN4NS5K6Q, seg_pos={0 0}, upload=58394]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:22.013095164 (60h54m31.735658214s) [noise=true , address=77.90.31.15:4697 , piece_id=DT7CTZTYNNEJPKHK435O3XVYZNJ2TI55GJMLCM4UVXN5IBYBZ5FA, seg_pos={0 0}, upload=58394]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:33604->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:33604->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33604->77.90.31.15:4697: use of closed network connection Jul 27 00:30:22.013122086 (60h54m31.735685136s) [noise=true , address=104.36.82.4:4801 , piece_id=ADCR6YLPHB7VXYCIXT6HLR3ACJ3FHBQLYKIPWOQWCBXPEOHQQBGA, seg_pos={0 0}, upload=58394]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:46330->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:22.013143317 (60h54m31.735706387s) [noise=true , address=77.90.31.11:4555 , piece_id=PXT2TZSUXDTKPIEWREAG224KZGJZ4333GG3PQ5V6PN3M4WTBMKBQ, seg_pos={0 0}, upload=58394]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:22.013204123 (60h54m31.735767193s) [noise=true , address=104.36.82.16:4293 , piece_id=AO42DKLBS6QAXDMJ4K2TKFX5RE4UTWDWKYI4B6HKYCVVB3WWD4CA, seg_pos={0 0}, upload=58394]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:42362->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:22.013251894 (60h54m31.735814944s) [seg_pos={0 0}, upload=58394]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:30:22.242105354 (60h54m31.964668394s) [upload=58394]: Done uploading segments... Jul 27 00:30:22.242133237 (60h54m31.964696277s) [upload=58394]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.230826955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX.] duration: 7.645276494s FAIL (2025-07-27 00:30:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 7.645302825s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X....X.X....XXX.X...X......X.XX...XXXX.....X..........X..X..] duration: 7.64924155s FAIL (2025-07-27 00:30:02 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.649258583s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235400757s FAIL (2025-07-27 00:29:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 8377]: Uploading piece... Jul 27 00:28:46.711272766 (60h52m56.433835806s) [noise=true , address=77.90.31.15:4721 , piece_id=HPMTWFFN6P7ZVSY7YC3QZSZ262BG34RR7QHLGRMTDS73UIMVZRBQ, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.711650058 (60h52m56.434213098s) [noise=true , address=79.127.201.215:4343 , piece_id=67FFLJWPZS67MRE32UIZ57AQXTDSUXNJ2W2T7HWNAQC44YZQG7OA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.711999758 (60h52m56.434562798s) [noise=true , address=104.36.82.14:4253 , piece_id=NHGD2RHKC57F53LMVMLZ7WBJ6ZRRSMZTLFNYBT4TKKQXZSC2TC7Q, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.712303679 (60h52m56.434866719s) [noise=true , address=79.127.205.245:4250 , piece_id=KLHAXR77NJLF4Y3TZJH23BEFZOKTK5FOFLXJNTWF4RZDWQJDX4HA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.712564868 (60h52m56.435127908s) [noise=true , address=77.90.31.17:4780 , piece_id=UGYIRQSDALEK6UHYB5AWHQ5OE67LDQIVYWBFRLRQMH6FWRD5S5EQ, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.712795831 (60h52m56.435358871s) [noise=true , address=104.36.82.53:4128 , piece_id=7BPHJUQKVUPTFEWGLR4M3SXLBMZ7AI6PZK3D73XRFJHSVR64PKMQ, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.713093891 (60h52m56.435656931s) [noise=true , address=79.127.205.233:4640 , piece_id=3OSVXOSUOFWHBYAFAADMYB5I5BJYZMEPI7L27L2WMFSC425DKOHA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.713347416 (60h52m56.435910456s) [noise=true , address=45.91.116.9:4277 , piece_id=PMOIKM7MMDDZWPORYSCBB74QWP2RP7T4RD6M5QD7KLLE563AVCTQ, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.713594198 (60h52m56.436157238s) [noise=true , address=104.36.82.52:4108 , piece_id=CVBA7Z5FR3GIANFC45ZWPRBY4Z7PIE7O3PP4UAGCKS4BQ6V4MHNA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.713864205 (60h52m56.436427245s) [noise=true , address=104.36.82.55:4197 , piece_id=PMZZPS6CJ6GCGFUNXHYYE3SEZ3S7JO4E6PT7MQWG7T6C56H576SA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.714178506 (60h52m56.436741536s) [noise=true , address=104.36.82.46:4503 , piece_id=LHAAZC3BEQR4UWUF2ZROVP36PUU73DBS257DQMRLYSFWX62MWVAA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.714431810 (60h52m56.43699485s) [noise=true , address=77.90.31.16:4743 , piece_id=ZFQRHC7UVDI2RSWJTMKQ5XNPDKHS3YS35DZPI53WUQMEE3QJHQHA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.714670818 (60h52m56.437233858s) [noise=true , address=64.49.13.138:4482 , piece_id=TL7QXM5NTLREIZ6XZ3EFFD6M7T7BNSG4IVSQBL4IT5SLURKF6ACA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.715522918 (60h52m56.438085948s) [noise=true , address=79.127.205.235:4690 , piece_id=J5OMFH6HJGEVFXOC67ZOSVJZSBTF3BUR5R2QVZ67VYIYQ7A6SO5Q, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:46.715770181 (60h52m56.438333221s) [noise=true , address=79.127.219.38:4017 , piece_id=BNOC6NR3TITKYEXYZCRIRCAQL677QTNJJY4GHMZ4KFQKWDDR3JKA, seg_pos={0 0}, upload=58377]: Uploading piece... Jul 27 00:28:47.798958155 (60h52m57.521521185s) [noise=true , address=77.90.31.15:4721 , piece_id=HPMTWFFN6P7ZVSY7YC3QZSZ262BG34RR7QHLGRMTDS73UIMVZRBQ, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn, address:77.90.31.15:4721): protocol: storage node overloaded, request limit: 2000 Jul 27 00:28:47.800722480 (60h52m57.52328552s) [noise=true , address=77.90.31.12:4621 , piece_id=KIUWSLCHV5HAH5BBZ6PFCJVBFDUZM425G6FC3QA2GCOYPCRZQAAA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: storage node overloaded, request limit: 2000 Jul 27 00:28:47.815528690 (60h52m57.53809173s) [upload=58377]: Next returned nil segment Jul 27 00:28:47.815545141 (60h52m57.538108171s) [upload=58377]: Waiting for error group managing segments... Jul 27 00:28:48.101555706 (60h52m57.824118746s) [noise=true , address=79.127.205.234:4608 , piece_id=RSMIE75BH7MRWOYWFZZWT7UQFHTTFPULYG6MBAY3F4R4RH25BEIA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: <nil> Jul 27 00:28:48.117363039 (60h52m57.839926079s) [noise=true , address=79.127.205.235:4690 , piece_id=J5OMFH6HJGEVFXOC67ZOSVJZSBTF3BUR5R2QVZ67VYIYQ7A6SO5Q, seg_pos={0 0}, upload=58377]: Done uploading piece. err: <nil> Jul 27 00:28:48.129849609 (60h52m57.852412639s) [noise=true , address=79.127.205.233:4640 , piece_id=3OSVXOSUOFWHBYAFAADMYB5I5BJYZMEPI7L27L2WMFSC425DKOHA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: <nil> Jul 27 00:28:48.189088602 (60h52m57.911651632s) [noise=true , address=64.49.13.138:4482 , piece_id=TL7QXM5NTLREIZ6XZ3EFFD6M7T7BNSG4IVSQBL4IT5SLURKF6ACA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: <nil> Jul 27 00:28:48.213068062 (60h52m57.935631102s) [noise=true , address=45.91.116.9:4277 , piece_id=PMOIKM7MMDDZWPORYSCBB74QWP2RP7T4RD6M5QD7KLLE563AVCTQ, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:28:48.526260233 (60h52m58.248823273s) [noise=true , address=79.127.205.245:4250 , piece_id=KLHAXR77NJLF4Y3TZJH23BEFZOKTK5FOFLXJNTWF4RZDWQJDX4HA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: <nil> Jul 27 00:28:48.627462864 (60h52m58.350025904s) [noise=true , address=79.127.219.38:4017 , piece_id=BNOC6NR3TITKYEXYZCRIRCAQL677QTNJJY4GHMZ4KFQKWDDR3JKA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: <nil> Jul 27 00:29:06.829790417 (60h53m16.552353457s) [noise=true , address=79.127.201.215:4343 , piece_id=67FFLJWPZS67MRE32UIZ57AQXTDSUXNJ2W2T7HWNAQC44YZQG7OA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: <nil> Jul 27 00:29:45.612427409 (60h53m55.334990449s) [noise=true , address=104.36.82.14:4253 , piece_id=NHGD2RHKC57F53LMVMLZ7WBJ6ZRRSMZTLFNYBT4TKKQXZSC2TC7Q, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14, address:104.36.82.14:4253): protocol: rpc: dial tcp 104.36.82.14:4253: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:45.612532229 (60h53m55.335095269s) [noise=true , address=104.36.82.52:4108 , piece_id=CVBA7Z5FR3GIANFC45ZWPRBY4Z7PIE7O3PP4UAGCKS4BQ6V4MHNA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:12XJJZCUSHUS52YC6jhoPB4ayjsyKZT477PymPj4inkZtdUvheT, address:104.36.82.52:4108): protocol: rpc: dial tcp 104.36.82.52:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:45.612567316 (60h53m55.335130346s) [noise=true , address=77.90.31.17:4780 , piece_id=UGYIRQSDALEK6UHYB5AWHQ5OE67LDQIVYWBFRLRQMH6FWRD5S5EQ, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:1XfvmVdBudtFnKwNxCxuGbQMS6HSSkVwPkYmffa5X6e6uYNcuF, address:77.90.31.17:4780): protocol: read tcp 188.166.225.170:54018->77.90.31.17:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:45.612591222 (60h53m55.335154262s) [noise=true , address=104.36.82.53:4128 , piece_id=7BPHJUQKVUPTFEWGLR4M3SXLBMZ7AI6PZK3D73XRFJHSVR64PKMQ, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: write tcp 188.166.225.170:44532->104.36.82.53:4128: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:45.612686113 (60h53m55.335249153s) [noise=true , address=104.36.82.55:4197 , piece_id=PMZZPS6CJ6GCGFUNXHYYE3SEZ3S7JO4E6PT7MQWG7T6C56H576SA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:104.36.82.55:4197): protocol: rpc: dial tcp 104.36.82.55:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:45.612761338 (60h53m55.335324378s) [noise=true , address=77.90.31.16:4743 , piece_id=ZFQRHC7UVDI2RSWJTMKQ5XNPDKHS3YS35DZPI53WUQMEE3QJHQHA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, address:77.90.31.16:4743): protocol: rpc: dial tcp 77.90.31.16:4743: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:45.612782699 (60h53m55.335345739s) [noise=true , address=104.36.82.47:4540 , piece_id=DAX3KOH67IM3D346THSU5THSNMKQZQ3U4KMA7243BEPYKJGSTICA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:104.36.82.47:4540): protocol: rpc: dial tcp 104.36.82.47:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:45.612811414 (60h53m55.335374454s) [noise=true , address=104.36.82.46:4503 , piece_id=LHAAZC3BEQR4UWUF2ZROVP36PUU73DBS257DQMRLYSFWX62MWVAA, seg_pos={0 0}, upload=58377]: Done uploading piece. err: ecclient: upload failed (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm, address:104.36.82.46:4503): protocol: write tcp 188.166.225.170:53506->104.36.82.46:4503: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:45.612831541 (60h53m55.335394581s) [seg_pos={0 0}, upload=58377]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 27 00:29:45.846130043 (60h53m55.568693083s) [upload=58377]: Done uploading segments... Jul 27 00:29:45.846154379 (60h53m55.568717409s) [upload=58377]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.235469478s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.XX] duration: 30.232921317s FAIL (2025-07-27 00:29:42 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.232946877s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [X......X...XX..XX.X.XX...X...............XX......X...XXX..XX] duration: 1m0.000993634s FAIL (2025-07-27 00:29:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001007762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X..........................................X......] duration: 2.017639552s FAIL (2025-07-27 00:29:25 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.017662425s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXXX] duration: 30.233809814s FAIL (2025-07-27 00:29:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070670444 (60h52m25.793233484s) [noise=true , address=104.36.82.10:4077 , piece_id=DNXWQ2K4RPQBH7CBRGBNY3IYE34K5SOJUUR7563HC4VD4FG2LUQA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070695342 (60h52m25.793258382s) [seg_pos={0 0}, upload=58361]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:28:16.311840156 (60h52m26.034403206s) [upload=58361]: Done uploading segments... Jul 27 00:28:16.311867668 (60h52m26.034430719s) [upload=58361]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 27 00:28:44.591999571 (60h52m54.314562611s) [upload=58374]: Starting upload Jul 27 00:28:44.592052462 (60h52m54.314615502s) [upload=58374]: Done starting upload Jul 27 00:28:44.592228058 (60h52m54.314791098s) [upload=58374]: Uploading segments... Jul 27 00:28:44.592259758 (60h52m54.314822798s) [upload=58374]: Got next segment. Inline: false Jul 27 00:28:44.592281730 (60h52m54.31484477s) [upload=58374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:28:44.592285858 (60h52m54.314848898s) [upload=58374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:28:44.828464899 (60h52m54.551027939s) [seg_pos={0 0}, upload=58374]: Begin upload segment... Jul 27 00:28:44.828522289 (60h52m54.551085329s) [seg_pos={0 0}, upload=58374]: Done begin upload segment. Jul 27 00:28:44.828546966 (60h52m54.551110006s) [noise=true , address=45.91.116.11:4117 , piece_id=RKKXC72UC7PCU34L6G6BBWLJOFGQ346CNXQD7LEVX3ORVNTNLGOQ, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.828578948 (60h52m54.551141988s) [noise=true , address=45.91.116.13:4153 , piece_id=JFZFNR2HRKDLHRZVL2RGEIHDO764HN6L65I5CINGEJUOBMFRGNSQ, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.828888089 (60h52m54.551451119s) [noise=true , address=45.91.116.6:4974 , piece_id=X3BDZCDSHFBFJKLXNHHRCA3DXTLGBJL3ZTLOUZVW5HIM72LRCVDA, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.829044568 (60h52m54.551607608s) [noise=true , address=45.91.116.14:4190 , piece_id=PZI3W4CV5BPSBK2N5DXX5JM6TU6KBTOEQKYZZJZSVHQX2ORLUOEQ, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.829151703 (60h52m54.551714733s) [noise=true , address=104.36.82.14:4221 , piece_id=E4XQ5ASV6APGJAY56SKEEQFR22BIWXZIH5ECX2RTMTZDZH6CLYHA, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.829400960 (60h52m54.551964s) [noise=true , address=45.91.116.4:4937 , piece_id=BQMK2EDRO7Z26VPUVHCKIQNDSNL4RSUXJHXPIBMDMHOZLSAZYWJA, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.830312715 (60h52m54.552875745s) [noise=true , address=77.90.31.12:4592 , piece_id=DREP7KTKNSSVYIBLPJ6NCY6WJ4FKJN7H3B67RVFJUA4V3K2ZT2OA, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.830924305 (60h52m54.553487345s) [noise=true , address=77.90.31.15:4697 , piece_id=QQDPDWCV3O5MC7YLCCH4RC4IAGG4CZZ3WKHXTFIHMSXUCRPUZDTA, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.831164435 (60h52m54.553727475s) [noise=true , address=104.36.82.15:4259 , piece_id=FJRYGEYB77KMKPYCHNWDYOJH5GXFVRKHACWCXBJTEJFGGN6Q7AVQ, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.831405225 (60h52m54.553968265s) [noise=true , address=104.36.82.5:4838 , piece_id=ZQJ4T6IKM7EUDFT7SMWJ2PPARIMKI6HHIN5Z3ZLBNM3LPEILR2PA, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.831995495 (60h52m54.554558535s) [noise=true , address=45.91.116.5:4297 , piece_id=5OO73VRTU4GJLXEVIMHMYENQN2M5LMO6PPWIFC7I5SHTEJDKDPDA, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.832227989 (60h52m54.554791029s) [noise=true , address=77.90.31.10:4520 , piece_id=DR52FBH7LD4YOQVE4WXXTRHCWQQN4ZTFIL3XI7EIKA5WD254XMIQ, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.832444824 (60h52m54.555007854s) [noise=true , address=77.90.31.14:4662 , piece_id=LTHBKUVGUD65XZZLLSJYXUSCZNCDAKZXH4I6EV5O3TIUQ26KTWNQ, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.832647652 (60h52m54.555210682s) [noise=true , address=77.90.31.19:4843 , piece_id=C3Y46YZNOIW5LH2PCACMWDN3GIMYNDNP5OM3RHOTVMF4OOV2DMLQ, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.832862052 (60h52m54.555425092s) [noise=true , address=45.91.116.7:4009 , piece_id=W2ABMWJM4KVWOALTXO5AASOTBFG5EEIFHJA77277HV7STXXAZ2HA, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:44.833092333 (60h52m54.555655373s) [noise=true , address=104.36.82.16:4295 , piece_id=RRC63VBCQLLURXOSQSKN7VPFBX6MAGYVNW2L2AKREZT7QFCLYT3A, seg_pos={0 0}, upload=58374]: Uploading piece... Jul 27 00:28:46.567879391 (60h52m56.290442432s) [upload=58374]: Next returned nil segment Jul 27 00:28:46.567921533 (60h52m56.290484563s) [upload=58374]: Waiting for error group managing segments... Jul 27 00:28:46.859557640 (60h52m56.58212068s) [noise=true , address=45.91.116.4:4937 , piece_id=BQMK2EDRO7Z26VPUVHCKIQNDSNL4RSUXJHXPIBMDMHOZLSAZYWJA, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:46.925038915 (60h52m56.647601955s) [noise=true , address=45.91.116.6:4974 , piece_id=X3BDZCDSHFBFJKLXNHHRCA3DXTLGBJL3ZTLOUZVW5HIM72LRCVDA, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:46.947189906 (60h52m56.669752946s) [noise=true , address=45.91.116.13:4153 , piece_id=JFZFNR2HRKDLHRZVL2RGEIHDO764HN6L65I5CINGEJUOBMFRGNSQ, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:46.951344034 (60h52m56.673907074s) [noise=true , address=45.91.116.14:4190 , piece_id=PZI3W4CV5BPSBK2N5DXX5JM6TU6KBTOEQKYZZJZSVHQX2ORLUOEQ, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:46.984783554 (60h52m56.707346584s) [noise=true , address=45.91.116.5:4297 , piece_id=5OO73VRTU4GJLXEVIMHMYENQN2M5LMO6PPWIFC7I5SHTEJDKDPDA, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:46.988573646 (60h52m56.711136676s) [noise=true , address=77.90.31.14:4662 , piece_id=LTHBKUVGUD65XZZLLSJYXUSCZNCDAKZXH4I6EV5O3TIUQ26KTWNQ, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:47.025300185 (60h52m56.747863225s) [noise=true , address=45.91.116.7:4009 , piece_id=W2ABMWJM4KVWOALTXO5AASOTBFG5EEIFHJA77277HV7STXXAZ2HA, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:47.057579054 (60h52m56.780142094s) [noise=true , address=45.91.116.11:4117 , piece_id=RKKXC72UC7PCU34L6G6BBWLJOFGQ346CNXQD7LEVX3ORVNTNLGOQ, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:48.284570658 (60h52m58.007133698s) [noise=true , address=77.90.31.19:4843 , piece_id=C3Y46YZNOIW5LH2PCACMWDN3GIMYNDNP5OM3RHOTVMF4OOV2DMLQ, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:48.975836947 (60h52m58.698399987s) [noise=true , address=77.90.31.15:4697 , piece_id=QQDPDWCV3O5MC7YLCCH4RC4IAGG4CZZ3WKHXTFIHMSXUCRPUZDTA, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:28:50.267311737 (60h52m59.989874777s) [noise=true , address=77.90.31.12:4592 , piece_id=DREP7KTKNSSVYIBLPJ6NCY6WJ4FKJN7H3B67RVFJUA4V3K2ZT2OA, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:29:06.284041646 (60h53m16.006604686s) [noise=true , address=77.90.31.10:4520 , piece_id=DR52FBH7LD4YOQVE4WXXTRHCWQQN4ZTFIL3XI7EIKA5WD254XMIQ, seg_pos={0 0}, upload=58374]: Done uploading piece. err: <nil> Jul 27 00:29:13.948811773 (60h53m23.671374813s) [noise=true , address=104.36.82.15:4259 , piece_id=FJRYGEYB77KMKPYCHNWDYOJH5GXFVRKHACWCXBJTEJFGGN6Q7AVQ, seg_pos={0 0}, upload=58374]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:13.948882518 (60h53m23.671445558s) [noise=true , address=104.36.82.16:4295 , piece_id=RRC63VBCQLLURXOSQSKN7VPFBX6MAGYVNW2L2AKREZT7QFCLYT3A, seg_pos={0 0}, upload=58374]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:13.948915532 (60h53m23.671478572s) [noise=true , address=104.36.82.14:4221 , piece_id=E4XQ5ASV6APGJAY56SKEEQFR22BIWXZIH5ECX2RTMTZDZH6CLYHA, seg_pos={0 0}, upload=58374]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:13.948936541 (60h53m23.671499581s) [noise=true , address=104.36.82.5:4838 , piece_id=ZQJ4T6IKM7EUDFT7SMWJ2PPARIMKI6HHIN5Z3ZLBNM3LPEILR2PA, seg_pos={0 0}, upload=58374]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:55422->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:13.948958303 (60h53m23.671521343s) [seg_pos={0 0}, upload=58374]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:29:14.180912281 (60h53m23.903475321s) [upload=58374]: Done uploading segments... Jul 27 00:29:14.180949542 (60h53m23.903512582s) [upload=58374]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.233839369s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X....X.X....XXX.X...X......X.XX...XXXX.....X..........X..X.] duration: 17.867544543s FAIL (2025-07-27 00:29:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.867568759s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.X] duration: 30.232811868s FAIL (2025-07-27 00:28:43 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.232831326s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.238540304s FAIL (2025-07-27 00:28:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=77.90.31.13:4659 , piece_id=NZRBSSZVEKLDUKJB47TA77QEFZ5GVGJAGHA7CH3QCZ4KIK5IRHWQ, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.722113659 (60h51m48.444676699s) [noise=true , address=104.36.82.54:4181 , piece_id=MOMRZY5WTQN7BA7VNTL2MMYNSLDFGKVFHYBTYONSVCJLPQB4B7CA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.722161811 (60h51m48.444724851s) [noise=true , address=45.91.116.4:4964 , piece_id=HZ33J55E4NIGVAXCFEH6IYKCDVKWWAEBWPM5EQKK3L4TGEGT6G5Q, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.722418303 (60h51m48.444981343s) [noise=true , address=79.127.205.246:4275 , piece_id=JHTG62SBAUGKPOG2S5Q57H5IVZSZ4AL53AZSBJM2NLF6NEVQSOTA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.722427781 (60h51m48.444990821s) [noise=true , address=104.36.82.55:4210 , piece_id=EVQD3W7LZ6ZE22WYOLYWEBKSDFFLTFDTVQ4CRFCWAMA5B45AXJ2A, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.722681786 (60h51m48.445244826s) [noise=true , address=104.36.82.46:4498 , piece_id=5YJIJDWY62EP5B2NUKBF2UX7J2KVGHPZOYB6WT2W6FERLNXZL3UA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.722699210 (60h51m48.44526224s) [noise=true , address=64.49.13.26:4953 , piece_id=H37PPSMDWBKRF4LUKWBAGZRQG4423L2GQTGXOPUU323RRACJHLQA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.722975438 (60h51m48.445538468s) [noise=true , address=104.36.82.52:4092 , piece_id=HPHY4OA25YF5DA55RTMOIH2V33BXQ34ORVOE5Z7C4J2KCEVNVPXA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.722982141 (60h51m48.445545181s) [noise=true , address=79.127.205.235:4668 , piece_id=6F5PMSOXC6DK2YLFXJFYPRI4NFTUYSGDISXUX5MBA4L7XHOTU6OA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.723008101 (60h51m48.445571141s) [noise=true , address=77.90.31.21:4905 , piece_id=CJOB6OCGNGQXPPXELLMKP22LMZJIJZAJ3U4FZBI7526CMAH7FABA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.723294879 (60h51m48.445857909s) [noise=true , address=104.36.82.47:4518 , piece_id=OFRKVCFF7I42OTYMUGXMPQJKTMJWX2XBQ2TWEJIRIPP7H3VPJVBA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.724096013 (60h51m48.446659053s) [noise=true , address=77.90.31.16:4761 , piece_id=SSQPNG7UZS6GDFPWQLHTUIXBN44ZBBMN2QIGXEXYRVTFLGGMEAKA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.724377000 (60h51m48.44694004s) [noise=true , address=79.127.219.38:4005 , piece_id=DR5TRODJ5CVJ4SQCEHICJP3UMDZVDMIHSNSRFEXJZAXZA6CNOP2A, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.724971457 (60h51m48.447534487s) [noise=true , address=79.127.205.234:4616 , piece_id=FGIQTFKQHUEHCHBJ2N6BRGXFOI36QO46VKL3QIUDUVMMVRJ5ZTTQ, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.725395689 (60h51m48.447958729s) [noise=true , address=64.49.13.138:4488 , piece_id=FPCV3SLHC5KPBO5LIUB4WTKE3HIK6SVPFLVSB7C5ZC4I22FDALUQ, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.729229584 (60h51m48.451792614s) [noise=true , address=104.36.82.56:4235 , piece_id=Z4F3TG2WMMHBCDHNZOGZQP6UT25R4NQ6ZPZQWCLMKN4TFOXHQYEA, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:38.731118058 (60h51m48.453681098s) [noise=true , address=104.36.82.5:4866 , piece_id=RR6542LBU2TDSMU7UKHK45HYA4R2PLHZTBOUQ266OIJW4WQMCPRQ, seg_pos={0 0}, upload=58359]: Uploading piece... Jul 27 00:27:39.827220762 (60h51m49.549783802s) [upload=58359]: Next returned nil segment Jul 27 00:27:39.827239178 (60h51m49.549802218s) [upload=58359]: Waiting for error group managing segments... Jul 27 00:27:40.129200430 (60h51m49.85176347s) [noise=true , address=79.127.205.235:4668 , piece_id=6F5PMSOXC6DK2YLFXJFYPRI4NFTUYSGDISXUX5MBA4L7XHOTU6OA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:27:40.170677248 (60h51m49.893240288s) [noise=true , address=79.127.205.246:4275 , piece_id=JHTG62SBAUGKPOG2S5Q57H5IVZSZ4AL53AZSBJM2NLF6NEVQSOTA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:27:40.437354773 (60h51m50.159917813s) [noise=true , address=79.127.205.234:4616 , piece_id=FGIQTFKQHUEHCHBJ2N6BRGXFOI36QO46VKL3QIUDUVMMVRJ5ZTTQ, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:27:40.770782791 (60h51m50.493345831s) [noise=true , address=77.90.31.21:4905 , piece_id=CJOB6OCGNGQXPPXELLMKP22LMZJIJZAJ3U4FZBI7526CMAH7FABA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:27:41.162195952 (60h51m50.884758992s) [noise=true , address=79.127.219.38:4005 , piece_id=DR5TRODJ5CVJ4SQCEHICJP3UMDZVDMIHSNSRFEXJZAXZA6CNOP2A, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:27:42.093173773 (60h51m51.815736813s) [noise=true , address=77.90.31.12:4597 , piece_id=6ZQJO4W2HQKNYJV5WQRWZLRULTS4SDA45NDQ7PKHRDUC2LF7IBOQ, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:27:42.394361147 (60h51m52.116924187s) [noise=true , address=45.91.116.4:4964 , piece_id=HZ33J55E4NIGVAXCFEH6IYKCDVKWWAEBWPM5EQKK3L4TGEGT6G5Q, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:27:43.233840865 (60h51m52.956403905s) [noise=true , address=64.49.13.26:4953 , piece_id=H37PPSMDWBKRF4LUKWBAGZRQG4423L2GQTGXOPUU323RRACJHLQA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:27:43.253321728 (60h51m52.975884768s) [noise=true , address=64.49.13.138:4488 , piece_id=FPCV3SLHC5KPBO5LIUB4WTKE3HIK6SVPFLVSB7C5ZC4I22FDALUQ, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:12mQqzHsTJvR5wHpEGXDixk3NcRuBKxY2cL82eJMHidcxFiPCZb, address:64.49.13.138:4488): protocol: storage node overloaded, request limit: 2000 Jul 27 00:28:11.280211356 (60h52m21.002774396s) [noise=true , address=77.90.31.13:4659 , piece_id=NZRBSSZVEKLDUKJB47TA77QEFZ5GVGJAGHA7CH3QCZ4KIK5IRHWQ, seg_pos={0 0}, upload=58359]: Done uploading piece. err: <nil> Jul 27 00:28:37.631488971 (60h52m47.354052001s) [noise=true , address=104.36.82.56:4235 , piece_id=Z4F3TG2WMMHBCDHNZOGZQP6UT25R4NQ6ZPZQWCLMKN4TFOXHQYEA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:1a1sW1G1WPL5WQ2XxprWKQMsJkcb5ETSkk3ULp1TjRaZWptaT, address:104.36.82.56:4235): protocol: write tcp 188.166.225.170:53922->104.36.82.56:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:37.631531352 (60h52m47.354094392s) [noise=true , address=104.36.82.47:4518 , piece_id=OFRKVCFF7I42OTYMUGXMPQJKTMJWX2XBQ2TWEJIRIPP7H3VPJVBA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:104.36.82.47:4518): protocol: rpc: dial tcp 104.36.82.47:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:37.631562531 (60h52m47.354125572s) [noise=true , address=104.36.82.54:4181 , piece_id=MOMRZY5WTQN7BA7VNTL2MMYNSLDFGKVFHYBTYONSVCJLPQB4B7CA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:12gGf5E7tzF2TXtLWgpK55t24SC1BpTSeC9G66nPmR1ARnDrAE8, address:104.36.82.54:4181): protocol: rpc: dial tcp 104.36.82.54:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:37.631578252 (60h52m47.354141292s) [noise=true , address=104.36.82.55:4210 , piece_id=EVQD3W7LZ6ZE22WYOLYWEBKSDFFLTFDTVQ4CRFCWAMA5B45AXJ2A, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: rpc: dial tcp 104.36.82.55:4210: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:37.631612417 (60h52m47.354175457s) [noise=true , address=104.36.82.46:4498 , piece_id=5YJIJDWY62EP5B2NUKBF2UX7J2KVGHPZOYB6WT2W6FERLNXZL3UA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:1jyjxdWkBRBkDHSKknNmPdcnJ3HRcSCUQNnmpb1CaiEdMPEGk4, address:104.36.82.46:4498): protocol: write tcp 188.166.225.170:47506->104.36.82.46:4498: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:37.631629029 (60h52m47.354192059s) [noise=true , address=77.90.31.16:4761 , piece_id=SSQPNG7UZS6GDFPWQLHTUIXBN44ZBBMN2QIGXEXYRVTFLGGMEAKA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:37.631634750 (60h52m47.35419778s) [noise=true , address=104.36.82.52:4092 , piece_id=HPHY4OA25YF5DA55RTMOIH2V33BXQ34ORVOE5Z7C4J2KCEVNVPXA, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:12wYaUu5QpFU8T4mZt1iqeHqVJKJ5CyVW5vkeTWf1a8G3pLsx3P, address:104.36.82.52:4092): protocol: rpc: dial tcp 104.36.82.52:4092: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:37.631642104 (60h52m47.354205144s) [noise=true , address=104.36.82.5:4866 , piece_id=RR6542LBU2TDSMU7UKHK45HYA4R2PLHZTBOUQ266OIJW4WQMCPRQ, seg_pos={0 0}, upload=58359]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:104.36.82.5:4866): protocol: write tcp 188.166.225.170:51822->104.36.82.5:4866: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:37.631663766 (60h52m47.354226806s) [seg_pos={0 0}, upload=58359]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:28:37.868103125 (60h52m47.590666165s) [upload=58359]: Done uploading segments... Jul 27 00:28:37.868129075 (60h52m47.590692115s) [upload=58359]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.238555292s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X..........................................X.....] duration: 1.822820241s FAIL (2025-07-27 00:28:27 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.822847604s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.X......X...XX..XX.X.XX...X...............XX......X...XXX..X] duration: 1m0.001143858s FAIL (2025-07-27 00:28:17 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001168375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XXX] duration: 30.24284958s FAIL (2025-07-27 00:28:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 7 00:27:46.719859324 (60h51m56.442422364s) [upload=58361]: Done starting upload Jul 27 00:27:46.720002880 (60h51m56.44256592s) [upload=58361]: Uploading segments... Jul 27 00:27:46.720143799 (60h51m56.442706859s) [upload=58361]: Got next segment. Inline: false Jul 27 00:27:46.720191270 (60h51m56.4427543s) [upload=58361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:27:46.720196781 (60h51m56.442759821s) [upload=58361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:27:47.192266913 (60h51m56.914829953s) [seg_pos={0 0}, upload=58361]: Begin upload segment... Jul 27 00:27:47.192315045 (60h51m56.914878085s) [seg_pos={0 0}, upload=58361]: Done begin upload segment. Jul 27 00:27:47.192332157 (60h51m56.914895197s) [noise=true , address=45.91.116.12:4225 , piece_id=66HOLX2FMX4U4RSLGYKCROD425XUIXJHFQYUEUN47BUOGIE6DQBA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.192339902 (60h51m56.914902942s) [noise=true , address=45.91.116.13:4153 , piece_id=5E7NKTJPDHGH7IWYSFE4CRNFWYLTJKRPGOCZEI653FR3DV3FQLVQ, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.192703989 (60h51m56.915267029s) [noise=true , address=104.36.82.5:4837 , piece_id=N3KD6RHD7AXCXNF7TRN4IVLT6JK7FPEX4PNN6BQ33I6PLLOU2HFA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.192712776 (60h51m56.915275806s) [noise=true , address=77.90.31.21:4878 , piece_id=GNROAAE6XLCVA7LITD6HNYA2BSBRH7VZSXMJCDPICV6L3C7VDTVA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.192949249 (60h51m56.915512289s) [noise=true , address=104.36.82.11:4113 , piece_id=C4BTLU4X3PNKVPTFGN6WMSMZ5BSIK5LEGGO7765NSWASR5BRHKNA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.193188566 (60h51m56.915751606s) [noise=true , address=45.91.116.11:4118 , piece_id=45AMYZVDBEZSLUZZ7PEMTOXZDAH4DIBDXWPBNXDDIVAMBL46Q4RQ, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.193512175 (60h51m56.916075215s) [noise=true , address=104.36.82.15:4257 , piece_id=UMYK3FRUKCRPI55OCWKZTPTKOP2UZGVSSWCWYDSJ4N5SVPLLS4KA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.194339288 (60h51m56.916902318s) [noise=true , address=77.90.31.12:4590 , piece_id=MOQ2LPYGMUS24TIPDPWWVK5LFTOA7BVF5D5CR23WGXYPAURC3N3A, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.194627660 (60h51m56.9171907s) [noise=true , address=45.91.116.14:4190 , piece_id=WNMKGCRLIYUS2GU37HOWO4I3KTPEBIIUR64K6MZJSEHLNTEBH7ZQ, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.194868069 (60h51m56.917431109s) [noise=true , address=104.36.82.13:4187 , piece_id=M5YZD4PN5L3EKQE4EYPKPERJKLSVAGCYUOMVL64YWIQFE3AFEFIA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.195086838 (60h51m56.917649878s) [noise=true , address=104.36.82.12:4150 , piece_id=A76BRUD5B3CXB7E22VEB4XNVS3EBKCYMU4V4YXZ63C3R2NSJGVOQ, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.195306929 (60h51m56.917869969s) [noise=true , address=104.36.82.10:4077 , piece_id=DNXWQ2K4RPQBH7CBRGBNY3IYE34K5SOJUUR7563HC4VD4FG2LUQA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.195548311 (60h51m56.918111351s) [noise=true , address=77.90.31.19:4842 , piece_id=KAWHNT2EGJB5LHJAGOHLAZGASMNFWSCRRRJVC3HGRQO7ILIQAZJA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.195751811 (60h51m56.918314851s) [noise=true , address=77.90.31.10:4518 , piece_id=B66TNNWRDS5X62NBRVWOBPJ4NO3AULH5A5WMPDFDCGWIY3DJYSNA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.196038709 (60h51m56.918601749s) [noise=true , address=77.90.31.18:4805 , piece_id=S3RA6LIBKTSDOF2S4R52JMOFA6EQS5OBQ6ZAQMV27LPPFMC7OKHA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:47.196360034 (60h51m56.918923064s) [noise=true , address=104.36.82.9:4007 , piece_id=XFGF4XPGZYGI3QB2623ZIAIGNH7IJHWKM2DLXZ6B436HJEQ4DFVA, seg_pos={0 0}, upload=58361]: Uploading piece... Jul 27 00:27:48.933514472 (60h51m58.656077502s) [upload=58361]: Next returned nil segment Jul 27 00:27:48.933534531 (60h51m58.656097571s) [upload=58361]: Waiting for error group managing segments... Jul 27 00:27:49.218628530 (60h51m58.941191571s) [noise=true , address=45.91.116.13:4153 , piece_id=5E7NKTJPDHGH7IWYSFE4CRNFWYLTJKRPGOCZEI653FR3DV3FQLVQ, seg_pos={0 0}, upload=58361]: Done uploading piece. err: <nil> Jul 27 00:27:49.306065795 (60h51m59.028628835s) [noise=true , address=45.91.116.11:4118 , piece_id=45AMYZVDBEZSLUZZ7PEMTOXZDAH4DIBDXWPBNXDDIVAMBL46Q4RQ, seg_pos={0 0}, upload=58361]: Done uploading piece. err: <nil> Jul 27 00:27:49.373863643 (60h51m59.096426683s) [noise=true , address=45.91.116.12:4225 , piece_id=66HOLX2FMX4U4RSLGYKCROD425XUIXJHFQYUEUN47BUOGIE6DQBA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: <nil> Jul 27 00:27:49.401482766 (60h51m59.124045796s) [noise=true , address=45.91.116.14:4190 , piece_id=WNMKGCRLIYUS2GU37HOWO4I3KTPEBIIUR64K6MZJSEHLNTEBH7ZQ, seg_pos={0 0}, upload=58361]: Done uploading piece. err: <nil> Jul 27 00:27:49.403840978 (60h51m59.126404019s) [noise=true , address=77.90.31.21:4878 , piece_id=GNROAAE6XLCVA7LITD6HNYA2BSBRH7VZSXMJCDPICV6L3C7VDTVA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: <nil> Jul 27 00:27:50.618632772 (60h52m0.341195812s) [noise=true , address=77.90.31.12:4590 , piece_id=MOQ2LPYGMUS24TIPDPWWVK5LFTOA7BVF5D5CR23WGXYPAURC3N3A, seg_pos={0 0}, upload=58361]: Done uploading piece. err: <nil> Jul 27 00:27:50.737040051 (60h52m0.459603081s) [noise=true , address=77.90.31.18:4805 , piece_id=S3RA6LIBKTSDOF2S4R52JMOFA6EQS5OBQ6ZAQMV27LPPFMC7OKHA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: <nil> Jul 27 00:27:51.237771652 (60h52m0.960334692s) [noise=true , address=77.90.31.19:4842 , piece_id=KAWHNT2EGJB5LHJAGOHLAZGASMNFWSCRRRJVC3HGRQO7ILIQAZJA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: <nil> Jul 27 00:28:16.070387132 (60h52m25.792950172s) [noise=true , address=104.36.82.15:4257 , piece_id=UMYK3FRUKCRPI55OCWKZTPTKOP2UZGVSSWCWYDSJ4N5SVPLLS4KA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:45188->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070450563 (60h52m25.793013603s) [noise=true , address=104.36.82.11:4113 , piece_id=C4BTLU4X3PNKVPTFGN6WMSMZ5BSIK5LEGGO7765NSWASR5BRHKNA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:39334->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070517452 (60h52m25.793080482s) [noise=true , address=77.90.31.10:4518 , piece_id=B66TNNWRDS5X62NBRVWOBPJ4NO3AULH5A5WMPDFDCGWIY3DJYSNA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070579260 (60h52m25.7931423s) [noise=true , address=104.36.82.9:4007 , piece_id=XFGF4XPGZYGI3QB2623ZIAIGNH7IJHWKM2DLXZ6B436HJEQ4DFVA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:55948->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070606482 (60h52m25.793169522s) [noise=true , address=104.36.82.5:4837 , piece_id=N3KD6RHD7AXCXNF7TRN4IVLT6JK7FPEX4PNN6BQ33I6PLLOU2HFA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:52826->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070613044 (60h52m25.793176084s) [noise=true , address=104.36.82.13:4187 , piece_id=M5YZD4PN5L3EKQE4EYPKPERJKLSVAGCYUOMVL64YWIQFE3AFEFIA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070649525 (60h52m25.793212565s) [noise=true , address=104.36.82.12:4150 , piece_id=A76BRUD5B3CXB7E22VEB4XNVS3EBKCYMU4V4YXZ63C3R2NSJGVOQ, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070670444 (60h52m25.793233484s) [noise=true , address=104.36.82.10:4077 , piece_id=DNXWQ2K4RPQBH7CBRGBNY3IYE34K5SOJUUR7563HC4VD4FG2LUQA, seg_pos={0 0}, upload=58361]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:16.070695342 (60h52m25.793258382s) [seg_pos={0 0}, upload=58361]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:28:16.311840156 (60h52m26.034403206s) [upload=58361]: Done uploading segments... Jul 27 00:28:16.311867668 (60h52m26.034430719s) [upload=58361]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.242877011s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X....X.X....XXX.X...X......X.XX...XXXX.....X..........X..X] duration: 30.001300254s FAIL (2025-07-27 00:27:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001323318s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X.] duration: 8.329090276s FAIL (2025-07-27 00:27:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 8.329105196s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.249543571s FAIL (2025-07-27 00:27:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== [upload=58345]: Got next segment. Inline: false Jul 27 00:26:37.803303071 (60h50m47.525866102s) [upload=58345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:26:37.803307209 (60h50m47.525870249s) [upload=58345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:26:38.247618197 (60h50m47.970181238s) [seg_pos={0 0}, upload=58345]: Begin upload segment... Jul 27 00:26:38.247668043 (60h50m47.970231083s) [seg_pos={0 0}, upload=58345]: Done begin upload segment. Jul 27 00:26:38.247677382 (60h50m47.970240422s) [noise=true , address=79.127.219.47:4048 , piece_id=C46EDMKQ3YBPBDDY62DFAYEXOH3W7PRIOJYYKWV4CRDRYK3ZTQFA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.247686148 (60h50m47.970249188s) [noise=true , address=156.146.43.225:4448 , piece_id=CYDRG2LGU6DTZP5XD7XMKQW2B2ZMJNMK3OF2MNMFGFXAKFVP6WFA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.247759318 (60h50m47.970322358s) [noise=true , address=45.91.116.9:4277 , piece_id=WXAU5XNITBIABAUJ754IAS44VJKRHTWY43BTWODCDOTBWPOHDDPA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.248053851 (60h50m47.970616881s) [noise=true , address=77.90.31.19:4867 , piece_id=LBDWWQ44CQU6ADVQ5TLLZAGFECXS47CJ3J5U5TWT72MX55RGEUYQ, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.248317205 (60h50m47.970880245s) [noise=true , address=79.127.205.234:4610 , piece_id=D5VNQZ7HJ2CGZEJE6EO45L4OYWI5QZBXYXGEON7VQRQ4C4OM5GEA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.248573586 (60h50m47.971136626s) [noise=true , address=45.91.116.12:4231 , piece_id=TF7H6QABRIOBVQIMAOU7CDMOXTN4C53QLSP4RDBSKCVOXVQ5RBFQ, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.248832231 (60h50m47.971395261s) [noise=true , address=45.91.116.14:4206 , piece_id=56HED72YCLGJ4Z4WTHKK2OK4Y3M2IETGJLBAITDEIVCLP7XDCFFQ, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.249064495 (60h50m47.971627525s) [noise=true , address=77.90.31.11:4569 , piece_id=BJQ6OSLDCEVIYRWVXGSP5HXN5NX2GFT3DSIOJ3E7DARFRZCOCBHQ, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.249256532 (60h50m47.971819573s) [noise=true , address=79.127.219.38:4006 , piece_id=GALBWCTGXIRYXUBUI6GQRX7FRA27UYRLBY26SV25RDUL3MRTUBDQ, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.249457376 (60h50m47.972020417s) [noise=true , address=104.36.82.55:4218 , piece_id=EZD3JBEPP3WOB4DYN7LP5F6FZ2TDRLUQPMOIKFI3ZD52KUBSXVHQ, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.249648733 (60h50m47.972211773s) [noise=true , address=104.36.82.6:4892 , piece_id=QTMLBADCPSU2RI6PY6UXQOWH7ZOFY52YNKYVHEI4CE66L2BD5CFA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.249885095 (60h50m47.972448135s) [noise=true , address=77.90.31.21:4912 , piece_id=IEKAI3OWAJJRJZ5VEP2NFK5XIMRNT2LY7PMOQLFTVDZEFK3NYFZA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.250074488 (60h50m47.972637528s) [noise=true , address=77.90.31.15:4715 , piece_id=X664TG2GDMUCYM7JEJIEJ2DN3JET5UQRELC4QHL3R3R3EYEVQUTA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.250255283 (60h50m47.972818313s) [noise=true , address=79.127.205.245:4241 , piece_id=VNYMYPJKAPRUUSGJ2GCOU6TDUNHZX3EZUQ2KGQMTEOCE2RXAXNXA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.250433996 (60h50m47.972997026s) [noise=true , address=64.49.13.26:4953 , piece_id=LHXBN46BWHQHLUPO5AQ2O3QWKQ4V67A55LXIOM7SIIR2UMGGOZOA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.250623999 (60h50m47.973187039s) [noise=true , address=79.127.205.235:4679 , piece_id=ZPKL6OXQHJMW3KV6I2IT3N55ZO2EWV37KVA6IETE4AMSQQ2ZFWJA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.250864849 (60h50m47.973427889s) [noise=true , address=104.36.82.47:4551 , piece_id=LOG3MYYZZCTPVB2TPAJIPUL5DMSPLW2WZLCK7O2NMGU4XX7TYMAA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:38.251066686 (60h50m47.973629726s) [noise=true , address=104.36.82.46:4493 , piece_id=ZEP2LHC3SUJJSYTVMA52WQNWRJY6RVCCHGRKQZG4MNT6OD4KGOKA, seg_pos={0 0}, upload=58345]: Uploading piece... Jul 27 00:26:39.348414773 (60h50m49.070977813s) [upload=58345]: Next returned nil segment Jul 27 00:26:39.348433138 (60h50m49.070996169s) [upload=58345]: Waiting for error group managing segments... Jul 27 00:26:39.646504488 (60h50m49.369067518s) [noise=true , address=79.127.205.245:4241 , piece_id=VNYMYPJKAPRUUSGJ2GCOU6TDUNHZX3EZUQ2KGQMTEOCE2RXAXNXA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:39.649598598 (60h50m49.372161638s) [noise=true , address=79.127.205.234:4610 , piece_id=D5VNQZ7HJ2CGZEJE6EO45L4OYWI5QZBXYXGEON7VQRQ4C4OM5GEA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:39.854697454 (60h50m49.577260494s) [noise=true , address=79.127.205.235:4679 , piece_id=ZPKL6OXQHJMW3KV6I2IT3N55ZO2EWV37KVA6IETE4AMSQQ2ZFWJA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:40.167500830 (60h50m49.89006386s) [noise=true , address=77.90.31.15:4715 , piece_id=X664TG2GDMUCYM7JEJIEJ2DN3JET5UQRELC4QHL3R3R3EYEVQUTA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:40.208961507 (60h50m49.931524537s) [noise=true , address=77.90.31.21:4912 , piece_id=IEKAI3OWAJJRJZ5VEP2NFK5XIMRNT2LY7PMOQLFTVDZEFK3NYFZA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:40.321108979 (60h50m50.043672019s) [noise=true , address=79.127.219.38:4006 , piece_id=GALBWCTGXIRYXUBUI6GQRX7FRA27UYRLBY26SV25RDUL3MRTUBDQ, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:40.349425747 (60h50m50.071988777s) [noise=true , address=64.49.13.26:4953 , piece_id=LHXBN46BWHQHLUPO5AQ2O3QWKQ4V67A55LXIOM7SIIR2UMGGOZOA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: ecclient: upload failed (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE, address:64.49.13.26:4953): protocol: storage node overloaded, request limit: 2000 Jul 27 00:26:42.739609688 (60h50m52.462172728s) [noise=true , address=156.146.43.225:4448 , piece_id=CYDRG2LGU6DTZP5XD7XMKQW2B2ZMJNMK3OF2MNMFGFXAKFVP6WFA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:43.703533724 (60h50m53.426096764s) [noise=true , address=45.91.116.9:4277 , piece_id=WXAU5XNITBIABAUJ754IAS44VJKRHTWY43BTWODCDOTBWPOHDDPA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:44.317755234 (60h50m54.040318264s) [noise=true , address=79.127.219.47:4048 , piece_id=C46EDMKQ3YBPBDDY62DFAYEXOH3W7PRIOJYYKWV4CRDRYK3ZTQFA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:44.731430629 (60h50m54.453993679s) [noise=true , address=45.91.116.12:4231 , piece_id=TF7H6QABRIOBVQIMAOU7CDMOXTN4C53QLSP4RDBSKCVOXVQ5RBFQ, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:46.165149866 (60h50m55.887712906s) [noise=true , address=45.91.116.14:4206 , piece_id=56HED72YCLGJ4Z4WTHKK2OK4Y3M2IETGJLBAITDEIVCLP7XDCFFQ, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:26:52.233095689 (60h51m1.955658729s) [noise=true , address=77.90.31.19:4867 , piece_id=LBDWWQ44CQU6ADVQ5TLLZAGFECXS47CJ3J5U5TWT72MX55RGEUYQ, seg_pos={0 0}, upload=58345]: Done uploading piece. err: <nil> Jul 27 00:27:37.162236246 (60h51m46.884799286s) [noise=true , address=104.36.82.46:4493 , piece_id=ZEP2LHC3SUJJSYTVMA52WQNWRJY6RVCCHGRKQZG4MNT6OD4KGOKA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:37.162340596 (60h51m46.884903636s) [noise=true , address=77.90.31.11:4569 , piece_id=BJQ6OSLDCEVIYRWVXGSP5HXN5NX2GFT3DSIOJ3E7DARFRZCOCBHQ, seg_pos={0 0}, upload=58345]: Done uploading piece. err: ecclient: upload failed (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ, address:77.90.31.11:4569): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:27:37.162576166 (60h51m46.885139206s) [noise=true , address=104.36.82.6:4892 , piece_id=QTMLBADCPSU2RI6PY6UXQOWH7ZOFY52YNKYVHEI4CE66L2BD5CFA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: ecclient: upload failed (node:12MmZivyjXMMHcDgU3aG4HkWqBgY2QRf4Q1pkxJDF5QKSTQ7RXA, address:104.36.82.6:4892): protocol: write tcp 188.166.225.170:55824->104.36.82.6:4892: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:37.162678031 (60h51m46.885241071s) [noise=true , address=104.36.82.47:4551 , piece_id=LOG3MYYZZCTPVB2TPAJIPUL5DMSPLW2WZLCK7O2NMGU4XX7TYMAA, seg_pos={0 0}, upload=58345]: Done uploading piece. err: ecclient: upload failed (node:1UhJvSMtNqJDak4TM7eUgz1N7yCn4ChmRRTe7nXdffTNsS1pch, address:104.36.82.47:4551): protocol: rpc: dial tcp 104.36.82.47:4551: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:37.162800296 (60h51m46.885363336s) [noise=true , address=104.36.82.55:4218 , piece_id=EZD3JBEPP3WOB4DYN7LP5F6FZ2TDRLUQPMOIKFI3ZD52KUBSXVHQ, seg_pos={0 0}, upload=58345]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 188.166.225.170:37466->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:37.162819672 (60h51m46.885382702s) [seg_pos={0 0}, upload=58345]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:27:37.409519612 (60h51m47.132082652s) [upload=58345]: Done uploading segments... Jul 27 00:27:37.409543798 (60h51m47.132106839s) [upload=58345]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.249554743s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............X..........................................X....] duration: 1.879054092s FAIL (2025-07-27 00:27:32 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.87909398s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.XX] duration: 30.235077505s FAIL (2025-07-27 00:27:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:52158->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:51.900931630 (60h51m1.62349466s) [seg_pos={0 0}, upload=58339]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 27 00:26:52.128183621 (60h51m1.850746671s) [upload=58339]: Done uploading segments... Jul 27 00:26:52.128235581 (60h51m1.850798621s) [upload=58339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Jul 27 00:27:01.556098246 (60h51m11.278661286s) [upload=58351]: Starting upload Jul 27 00:27:01.556155134 (60h51m11.278718174s) [upload=58351]: Done starting upload Jul 27 00:27:01.556231841 (60h51m11.278794881s) [upload=58351]: Uploading segments... Jul 27 00:27:01.556321252 (60h51m11.278884292s) [upload=58351]: Got next segment. Inline: false Jul 27 00:27:01.556349727 (60h51m11.278912767s) [upload=58351]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:27:01.556354265 (60h51m11.278917305s) [upload=58351]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:27:01.791367055 (60h51m11.513930115s) [seg_pos={0 0}, upload=58351]: Begin upload segment... Jul 27 00:27:01.791443211 (60h51m11.514006261s) [seg_pos={0 0}, upload=58351]: Done begin upload segment. Jul 27 00:27:01.791462829 (60h51m11.514025869s) [noise=true , address=104.36.82.2:4730 , piece_id=2EJDAMNAO6Q346I3XR3FYVZMWSJA3MTIQCGCWOWS6SC73SZTXU2A, seg_pos={0 0}, upload=58351]: Uploading piece... Jul 27 00:27:01.791488658 (60h51m11.514051708s) [noise=true , address=45.91.116.9:4263 , piece_id=NB6ED7JJJF3O2SQYFKAFQBU7IX34CHKYTDKN6EVCW6OIAQYOXADA, seg_pos={0 0}, upload=58351]: Uploading piece... Jul 27 00:27:01.791822968 (60h51m11.514386008s) [noise=true , address=77.90.31.17:4772 , piece_id=X3UE6FN26BA7HDETFZF5VCNDCQ67REL2RQFDUKSS7EE3LGREM5LA, seg_pos={0 0}, upload=58351]: Uploading piece... Jul 27 00:27:01.791969347 (60h51m11.514532388s) [noise=true , address=45.91.116.14:4191 , piece_id=YWJLTMK3PC3P2ZRI7ON2K5L2MI2CIPWUZLYAHR42PZ7RRUH5ATJQ, seg_pos={0 0}, upload=58351]: Uploading piece... Jul 27 00:27:01.792143521 (60h51m11.514706571s) [noise=true , address=104.36.82.16:4294 , piece_id=R3BGFVK7NVXHNAGQH5PAPT4X2S2Y4WCGPJGINJUSFFLP6EIOAIKQ, seg_pos={0 0}, upload=58351]: Uploading piece... Jul 27 00:27:01.792252520 (60h51m11.51481556s) [noise=true , address=77.90.31.12:4589 , piece_id=QBMOS3QMA25RAOPGOERU54A7WYYZY4CKZDWZHKCOEETKPD3KPOWQ, seg_pos={0 0}, upload=58351]: Uploading piece... Jul 27 00:27:01.792446591 (60h51m11.515009631s) [noise=true , address=45.91.116.5:4297 , piece_id=5HPYHKDIUGWNALVWRBCX4INTPFWHWCS7TDWVJWFCYH3VBVJLBIOQ, seg_pos={0 0}, upload=58351]: Uploading piece... Jul 27 00:27:01.792683153 (60h51m11.515246193s) [noise=true , address=77.90.31.15:4698 , piece_id=EQR733OXLMPGOUSLDBAJN5IBGROE2YMVRGNU5X6PVCYTWJMWPTRA, seg_pos={0 0}, upload=58351]: Uploading piece... Jul 27 00:27:01.792892314 (60h51m11.515455354s) [noise=true , address=45.91.116.12:4226 , piece_id=5RQGQGZ7L2P2KI5QI2G62ENBE65VVNKIOEQ4JNTKFKTLBLNXYVAA, seg_pos={0 0}, upload=58351]: Uploading piece... Jul 27 00:27:01.793106133 (60h51m11.515669173s) [noise=true , address=77.90.31.19:4842 , piece_id=2OTDFMGNTN6NIP7W3IITPS3S5XLRD35JPNSMO2OYCI4KDURI74TQ, seg_pos={0 0}, upload=58351]: Uploading piece... Jul 27 00:27:01.793337586 (60h51m11.515900626s) [noise=true , address=77.90.31.19:4841 , piece_id=SLILZE62LAAMO5S2FLFJSCL5ARNHYZNSPYNFHWCQJ3QFM4DKOXYQ, seg_pos={0 0}, upload=58351]: Uploading piece... Jul 27 00:27:01.793648781 (60h51m11.516211821s) [noise=true , address=104.36.82.3:4765 , piece_id=2XKYHOMDC7PSDS7C2E4TP2QWBLNVAQZYW4TJ5OAPLOZHS47IAWTA, seg_pos={0 0}, upload=58351]: Uploading piece... Jul 27 00:27:01.793947252 (60h51m11.516510292s) [noise=true , address=45.91.116.12:4225 , piece_id=Y2NFQ6LV4SEHCMWBFS3FBHNRTV4HILVFFSG73PNOTG7RT4BW4IRQ, seg_pos={0 0}, upload=58351]: Uploading piece... Jul 27 00:27:01.794178505 (60h51m11.516741555s) [noise=true , address=45.91.116.13:4155 , piece_id=INIZQIK7PV2MMCJTLXFAHPYUTC3KBT6VIMK5LA4BCZ3O3U5JKLMA, seg_pos={0 0}, upload=58351]: Uploading piece... Jul 27 00:27:01.794389789 (60h51m11.516952829s) [noise=true , address=45.91.116.8:4047 , piece_id=3V4WREDHULXVU4EJCHWDUUEYVS2GRQK5LFJOT4XF3KUVNG7QH6HQ, seg_pos={0 0}, upload=58351]: Uploading piece... Jul 27 00:27:01.794603678 (60h51m11.517166718s) [noise=true , address=104.36.82.11:4113 , piece_id=QVRPWOVOXEHX4WFI5ZLMSTKL3LBDPE3SJYGMWCGGXK3BCXICKTUA, seg_pos={0 0}, upload=58351]: Uploading piece... Jul 27 00:27:03.530051332 (60h51m13.252614362s) [upload=58351]: Next returned nil segment Jul 27 00:27:03.530070047 (60h51m13.252633087s) [upload=58351]: Waiting for error group managing segments... Jul 27 00:27:03.806407393 (60h51m13.528970433s) [noise=true , address=45.91.116.8:4047 , piece_id=3V4WREDHULXVU4EJCHWDUUEYVS2GRQK5LFJOT4XF3KUVNG7QH6HQ, seg_pos={0 0}, upload=58351]: Done uploading piece. err: <nil> Jul 27 00:27:03.812862033 (60h51m13.535425073s) [noise=true , address=77.90.31.15:4698 , piece_id=EQR733OXLMPGOUSLDBAJN5IBGROE2YMVRGNU5X6PVCYTWJMWPTRA, seg_pos={0 0}, upload=58351]: Done uploading piece. err: <nil> Jul 27 00:27:03.821390982 (60h51m13.543954022s) [noise=true , address=45.91.116.12:4225 , piece_id=Y2NFQ6LV4SEHCMWBFS3FBHNRTV4HILVFFSG73PNOTG7RT4BW4IRQ, seg_pos={0 0}, upload=58351]: Done uploading piece. err: <nil> Jul 27 00:27:03.884441446 (60h51m13.607004486s) [noise=true , address=45.91.116.9:4263 , piece_id=NB6ED7JJJF3O2SQYFKAFQBU7IX34CHKYTDKN6EVCW6OIAQYOXADA, seg_pos={0 0}, upload=58351]: Done uploading piece. err: <nil> Jul 27 00:27:03.885019712 (60h51m13.607582752s) [noise=true , address=45.91.116.5:4297 , piece_id=5HPYHKDIUGWNALVWRBCX4INTPFWHWCS7TDWVJWFCYH3VBVJLBIOQ, seg_pos={0 0}, upload=58351]: Done uploading piece. err: <nil> Jul 27 00:27:03.895743731 (60h51m13.618306771s) [noise=true , address=45.91.116.13:4155 , piece_id=INIZQIK7PV2MMCJTLXFAHPYUTC3KBT6VIMK5LA4BCZ3O3U5JKLMA, seg_pos={0 0}, upload=58351]: Done uploading piece. err: <nil> Jul 27 00:27:03.909907310 (60h51m13.63247035s) [noise=true , address=45.91.116.12:4226 , piece_id=5RQGQGZ7L2P2KI5QI2G62ENBE65VVNKIOEQ4JNTKFKTLBLNXYVAA, seg_pos={0 0}, upload=58351]: Done uploading piece. err: <nil> Jul 27 00:27:03.942838798 (60h51m13.665401828s) [noise=true , address=45.91.116.14:4191 , piece_id=YWJLTMK3PC3P2ZRI7ON2K5L2MI2CIPWUZLYAHR42PZ7RRUH5ATJQ, seg_pos={0 0}, upload=58351]: Done uploading piece. err: <nil> Jul 27 00:27:05.194602321 (60h51m14.917165361s) [noise=true , address=77.90.31.19:4841 , piece_id=SLILZE62LAAMO5S2FLFJSCL5ARNHYZNSPYNFHWCQJ3QFM4DKOXYQ, seg_pos={0 0}, upload=58351]: Done uploading piece. err: <nil> Jul 27 00:27:05.318649279 (60h51m15.041212319s) [noise=true , address=77.90.31.12:4589 , piece_id=QBMOS3QMA25RAOPGOERU54A7WYYZY4CKZDWZHKCOEETKPD3KPOWQ, seg_pos={0 0}, upload=58351]: Done uploading piece. err: <nil> Jul 27 00:27:05.537154344 (60h51m15.259717374s) [noise=true , address=77.90.31.19:4842 , piece_id=2OTDFMGNTN6NIP7W3IITPS3S5XLRD35JPNSMO2OYCI4KDURI74TQ, seg_pos={0 0}, upload=58351]: Done uploading piece. err: <nil> Jul 27 00:27:30.913457670 (60h51m40.6360207s) [noise=true , address=104.36.82.3:4765 , piece_id=2XKYHOMDC7PSDS7C2E4TP2QWBLNVAQZYW4TJ5OAPLOZHS47IAWTA, seg_pos={0 0}, upload=58351]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:30.913497637 (60h51m40.636060677s) [noise=true , address=104.36.82.16:4294 , piece_id=R3BGFVK7NVXHNAGQH5PAPT4X2S2Y4WCGPJGINJUSFFLP6EIOAIKQ, seg_pos={0 0}, upload=58351]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:30.913508227 (60h51m40.636071267s) [noise=true , address=104.36.82.11:4113 , piece_id=QVRPWOVOXEHX4WFI5ZLMSTKL3LBDPE3SJYGMWCGGXK3BCXICKTUA, seg_pos={0 0}, upload=58351]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:57354->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:30.913514730 (60h51m40.63607777s) [noise=true , address=77.90.31.17:4772 , piece_id=X3UE6FN26BA7HDETFZF5VCNDCQ67REL2RQFDUKSS7EE3LGREM5LA, seg_pos={0 0}, upload=58351]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:41912->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:30.913534647 (60h51m40.636097688s) [noise=true , address=104.36.82.2:4730 , piece_id=2EJDAMNAO6Q346I3XR3FYVZMWSJA3MTIQCGCWOWS6SC73SZTXU2A, seg_pos={0 0}, upload=58351]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:36822->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:30.913556970 (60h51m40.63612001s) [seg_pos={0 0}, upload=58351]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:27:31.146384660 (60h51m40.86894779s) [upload=58351]: Done uploading segments... Jul 27 00:27:31.146422322 (60h51m40.868985452s) [upload=58351]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.235100629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.X] duration: 30.234665215s FAIL (2025-07-27 00:27:30 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.234688451s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............X..........................................X...] duration: 1.825798041s FAIL (2025-07-27 00:27:01 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.825831996s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [..X......X...XX..XX.X.XX...X...............XX......X...XXX..] duration: 26.950358806s FAIL (2025-07-27 00:26:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.950375007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.X] duration: 30.230348293s FAIL (2025-07-27 00:26:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== 2.502150606s) [seg_pos={0 0}, upload=58339]: Begin upload segment... Jul 27 00:26:22.779645407 (60h50m32.502208437s) [seg_pos={0 0}, upload=58339]: Done begin upload segment. Jul 27 00:26:22.779657720 (60h50m32.50222076s) [noise=true , address=45.91.116.4:4939 , piece_id=4QOEIM7P7HMKUGAML3CTRGXQMZDD3UOTKX6LDY6RQUEFNHBWCGOQ, seg_pos={0 0}, upload=58339]: Uploading piece... Jul 27 00:26:22.779668501 (60h50m32.502231541s) [noise=true , address=104.36.82.2:4732 , piece_id=OPIC7W4522NQCIFVBYVIDYF7T7CHTZHCTHW6YJ6FU3ZGOHX3QPLQ, seg_pos={0 0}, upload=58339]: Uploading piece... Jul 27 00:26:22.779749937 (60h50m32.502312977s) [noise=true , address=104.36.82.4:4803 , piece_id=2LM2HXDM5HT4WSJHFLPN3DQGPYPXUMOSK6DR7R22XWNZHG4NKBOQ, seg_pos={0 0}, upload=58339]: Uploading piece... Jul 27 00:26:22.780099305 (60h50m32.502662345s) [noise=true , address=104.36.82.14:4222 , piece_id=6AJPVV7ZU23Q7PDMVGNZ4Y6DT32ICHIQDMTNDJLTMZ7T4P2D3FMQ, seg_pos={0 0}, upload=58339]: Uploading piece... Jul 27 00:26:22.780154751 (60h50m32.502717791s) [noise=true , address=45.91.116.13:4154 , piece_id=OEMYIWAH2CZWB5ATTRTCSLHSEQCMQQIQJSWYDPZ4AHIWMJBYY7MA, seg_pos={0 0}, upload=58339]: Uploading piece... Jul 27 00:26:22.780467209 (60h50m32.503030249s) [noise=true , address=104.36.82.12:4150 , piece_id=QIO2OVQF7IGJ5KHI5HVJSCJG5UVZAEZI6YE2F3OPUZURU6IYJI7Q, seg_pos={0 0}, upload=58339]: Uploading piece... Jul 27 00:26:22.780740773 (60h50m32.503303813s) [noise=true , address=104.36.82.6:4873 , piece_id=25V6UMISU6KZDIWSUGY6IW7QHCBCLCYI7LME32SBKWIIXC5VIX5Q, seg_pos={0 0}, upload=58339]: Uploading piece... Jul 27 00:26:22.781033853 (60h50m32.503596883s) [noise=true , address=77.90.31.10:4518 , piece_id=WTTHLXYL4GSHJN2UJ4QUJMQRBI3BLLDBE3J4HZXLRRM4FXLIH47Q, seg_pos={0 0}, upload=58339]: Uploading piece... Jul 27 00:26:22.781425613 (60h50m32.503988653s) [noise=true , address=104.36.82.5:4838 , piece_id=FTZSFYMBWF7EMM4VBGWTHLLCTKFY7ASCEPM54YLQMYI3KYQ6EREA, seg_pos={0 0}, upload=58339]: Uploading piece... Jul 27 00:26:22.781713524 (60h50m32.504276564s) [noise=true , address=104.36.82.8:4971 , piece_id=AKJ23DOD53B7PN5YYW6FGXD7CW4TG4OFWRKHUSBH4SGVBIWMPUUA, seg_pos={0 0}, upload=58339]: Uploading piece... Jul 27 00:26:22.782008849 (60h50m32.504571889s) [noise=true , address=77.90.31.13:4628 , piece_id=XK3W7BJBYAUK5PZMDR56URGD3GHKI3QQIRZE76NDLDFUTIE6NAMA, seg_pos={0 0}, upload=58339]: Uploading piece... Jul 27 00:26:22.782270399 (60h50m32.504833439s) [noise=true , address=104.36.82.3:4767 , piece_id=QJ2MPYXTQBDY65Y6YPBUVBMTW5PO73QNC6LPDYJ7OEWQ53ZSMZNA, seg_pos={0 0}, upload=58339]: Uploading piece... Jul 27 00:26:22.782473989 (60h50m32.505037029s) [noise=true , address=104.36.82.9:4007 , piece_id=FVDV23X5US2NLK3LNFYRFZTO2U7NWYCRXF3OMCGAKY7BSJSRPGGA, seg_pos={0 0}, upload=58339]: Uploading piece... Jul 27 00:26:22.782759335 (60h50m32.505322375s) [noise=true , address=77.90.31.19:4843 , piece_id=AKLKV27UKSKHBQYTUK65WUVNH5SGZYGJI4BHA5UGT2KCQBRV5GXQ, seg_pos={0 0}, upload=58339]: Uploading piece... Jul 27 00:26:22.783023310 (60h50m32.50558634s) [noise=true , address=77.90.31.12:4590 , piece_id=PO7OQBPOTEOBP7WVBLEUW5QMRM24NR25RABMTEEPO5I6WADBW7CA, seg_pos={0 0}, upload=58339]: Uploading piece... Jul 27 00:26:22.783317472 (60h50m32.505880502s) [noise=true , address=77.90.31.10:4517 , piece_id=YLFC35OHMECPETG54DGZY66QCB5ACN3RKPR6ZMS4XKJRGLQLO6VQ, seg_pos={0 0}, upload=58339]: Uploading piece... Jul 27 00:26:24.527694189 (60h50m34.250257229s) [upload=58339]: Next returned nil segment Jul 27 00:26:24.527727394 (60h50m34.250290434s) [upload=58339]: Waiting for error group managing segments... Jul 27 00:26:24.960105202 (60h50m34.682668242s) [noise=true , address=45.91.116.13:4154 , piece_id=OEMYIWAH2CZWB5ATTRTCSLHSEQCMQQIQJSWYDPZ4AHIWMJBYY7MA, seg_pos={0 0}, upload=58339]: Done uploading piece. err: <nil> Jul 27 00:26:24.973153627 (60h50m34.695716667s) [noise=true , address=45.91.116.4:4939 , piece_id=4QOEIM7P7HMKUGAML3CTRGXQMZDD3UOTKX6LDY6RQUEFNHBWCGOQ, seg_pos={0 0}, upload=58339]: Done uploading piece. err: <nil> Jul 27 00:26:26.298905859 (60h50m36.021468899s) [noise=true , address=77.90.31.13:4628 , piece_id=XK3W7BJBYAUK5PZMDR56URGD3GHKI3QQIRZE76NDLDFUTIE6NAMA, seg_pos={0 0}, upload=58339]: Done uploading piece. err: <nil> Jul 27 00:26:26.846311200 (60h50m36.56887424s) [noise=true , address=77.90.31.12:4590 , piece_id=PO7OQBPOTEOBP7WVBLEUW5QMRM24NR25RABMTEEPO5I6WADBW7CA, seg_pos={0 0}, upload=58339]: Done uploading piece. err: <nil> Jul 27 00:26:27.051432479 (60h50m36.773995519s) [noise=true , address=77.90.31.19:4843 , piece_id=AKLKV27UKSKHBQYTUK65WUVNH5SGZYGJI4BHA5UGT2KCQBRV5GXQ, seg_pos={0 0}, upload=58339]: Done uploading piece. err: <nil> Jul 27 00:26:51.900566612 (60h51m1.623129652s) [noise=true , address=104.36.82.2:4732 , piece_id=OPIC7W4522NQCIFVBYVIDYF7T7CHTZHCTHW6YJ6FU3ZGOHX3QPLQ, seg_pos={0 0}, upload=58339]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:51.900612560 (60h51m1.6231756s) [noise=true , address=104.36.82.4:4803 , piece_id=2LM2HXDM5HT4WSJHFLPN3DQGPYPXUMOSK6DR7R22XWNZHG4NKBOQ, seg_pos={0 0}, upload=58339]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:51.900648398 (60h51m1.623211438s) [noise=true , address=104.36.82.6:4873 , piece_id=25V6UMISU6KZDIWSUGY6IW7QHCBCLCYI7LME32SBKWIIXC5VIX5Q, seg_pos={0 0}, upload=58339]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:51.900679698 (60h51m1.623242738s) [noise=true , address=104.36.82.14:4222 , piece_id=6AJPVV7ZU23Q7PDMVGNZ4Y6DT32ICHIQDMTNDJLTMZ7T4P2D3FMQ, seg_pos={0 0}, upload=58339]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:51.900710427 (60h51m1.623273467s) [noise=true , address=104.36.82.8:4971 , piece_id=AKJ23DOD53B7PN5YYW6FGXD7CW4TG4OFWRKHUSBH4SGVBIWMPUUA, seg_pos={0 0}, upload=58339]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:51.900744872 (60h51m1.623307913s) [noise=true , address=104.36.82.3:4767 , piece_id=QJ2MPYXTQBDY65Y6YPBUVBMTW5PO73QNC6LPDYJ7OEWQ53ZSMZNA, seg_pos={0 0}, upload=58339]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:51.900775521 (60h51m1.623338561s) [noise=true , address=104.36.82.12:4150 , piece_id=QIO2OVQF7IGJ5KHI5HVJSCJG5UVZAEZI6YE2F3OPUZURU6IYJI7Q, seg_pos={0 0}, upload=58339]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:51.900805609 (60h51m1.62336864s) [noise=true , address=77.90.31.10:4518 , piece_id=WTTHLXYL4GSHJN2UJ4QUJMQRBI3BLLDBE3J4HZXLRRM4FXLIH47Q, seg_pos={0 0}, upload=58339]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:51.900834705 (60h51m1.623397735s) [noise=true , address=104.36.82.5:4838 , piece_id=FTZSFYMBWF7EMM4VBGWTHLLCTKFY7ASCEPM54YLQMYI3KYQ6EREA, seg_pos={0 0}, upload=58339]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:51.900869742 (60h51m1.623432782s) [noise=true , address=77.90.31.10:4517 , piece_id=YLFC35OHMECPETG54DGZY66QCB5ACN3RKPR6ZMS4XKJRGLQLO6VQ, seg_pos={0 0}, upload=58339]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:51.900904829 (60h51m1.623467859s) [noise=true , address=104.36.82.9:4007 , piece_id=FVDV23X5US2NLK3LNFYRFZTO2U7NWYCRXF3OMCGAKY7BSJSRPGGA, seg_pos={0 0}, upload=58339]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:52158->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:51.900931630 (60h51m1.62349466s) [seg_pos={0 0}, upload=58339]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 27 00:26:52.128183621 (60h51m1.850746671s) [upload=58339]: Done uploading segments... Jul 27 00:26:52.128235581 (60h51m1.850798621s) [upload=58339]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.230374993s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.X....X.X....XXX.X...X......X.XX...XXXX.....X..........X..] duration: 6.188956484s FAIL (2025-07-27 00:26:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.188978175s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.241448601s FAIL (2025-07-27 00:26:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=58329]: Done begin upload segment. Jul 27 00:25:37.789169539 (60h49m47.511732579s) [noise=true , address=79.127.220.101:4041 , piece_id=O5LA5NBW7BOQKOYQV6HRI6ZXGSRYDAT3MB6DUU2LZ5ZR2IJ32KJA, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.789243069 (60h49m47.511806109s) [noise=true , address=79.127.201.215:4360 , piece_id=LTG5ARLINZPTM3AJOY7H46JD54HJ2AUX7G5IQJZK5RM4VMJFBUMQ, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.789515360 (60h49m47.51207839s) [noise=true , address=104.36.82.11:4127 , piece_id=YPWJAI2ZB4APTMDPZTZL2IEKBMSWEX5NS4XP32Y6M2ZZYZCAIWFQ, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.789783633 (60h49m47.512346673s) [noise=true , address=77.90.31.12:4600 , piece_id=NBAOBFIYWNQLY37FHRD4TCMRKB7GLRTAEPOK74CMRZKA2AXAQWZQ, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.789852394 (60h49m47.512415434s) [noise=true , address=104.36.82.54:4174 , piece_id=OEMAKOQW433D24T7A3W42ZZXTT3FWOTWPDXQCQA4WLVJ4K6MFCJA, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.790074280 (60h49m47.51263731s) [noise=true , address=79.127.205.246:4283 , piece_id=754EA6V7C5DCY2NB7EAQDQHA46GOPHULCSTGYKD6K7APXZHB74PA, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.790253963 (60h49m47.512817003s) [noise=true , address=104.36.82.46:4513 , piece_id=GIPIW45T3IXFFJ4GTIHLKGWD5WZPFAZEEF7VBFXX7FNZA7AMTUQQ, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.790327454 (60h49m47.512890494s) [noise=true , address=77.90.31.17:4793 , piece_id=O6SJMFD52DGD7Z5LDGMCAHVVM4CE45T4V37WMKC4COCQ7OVYJGWQ, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.790586650 (60h49m47.51314969s) [noise=true , address=79.127.220.98:4422 , piece_id=N7TLVBUMEXIY5SZDPZG22GPBHERKCNUMKR5EHY6I34RDQWI5C3CQ, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.790687172 (60h49m47.513250212s) [noise=true , address=77.90.31.16:4760 , piece_id=O7OA65X4GPKEJSSNL6CMDPKHXAHE2VPGD7WRUTFFUYMXBAVOPACQ, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.790798435 (60h49m47.513361475s) [noise=true , address=104.36.82.56:4230 , piece_id=IVAOPO5NSSH7RF634XKP3D2M4IGPQLDTX7EW3AFQ66TDGLBKHOZA, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.790996945 (60h49m47.513559985s) [noise=true , address=79.127.205.245:4223 , piece_id=XIWL4D7CYS463HTW5R7WHWDQTSS4SQKEFR2X42DRV7KEBQULHEWQ, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.791067480 (60h49m47.51363052s) [noise=true , address=79.127.205.233:4633 , piece_id=5BUNLCRAAZWZKBDWZ5EU76GZHPAOH3LBM3DGGIPMZ5E5G7FZIEIA, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.791207488 (60h49m47.513770528s) [noise=true , address=45.91.116.14:4194 , piece_id=XZHYYTFBGDOJQ6JZGMXYXWFGE7DJV7SH6XGNZSYT65OXCYOBEDHQ, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.791410135 (60h49m47.513973175s) [noise=true , address=45.91.116.9:4277 , piece_id=RQHX7NWXBCY3K2A43MKYGLVSA5CM3IQ543GWTW2XVNNDBYGX3HJQ, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.791481191 (60h49m47.514044232s) [noise=true , address=64.49.13.140:4508 , piece_id=5US2CRXFGHOAZH7EX5ZZLF4WTCM5DVVBU6TUN424CUEMW7LFDB6A, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.791873352 (60h49m47.514436392s) [noise=true , address=79.127.205.235:4691 , piece_id=ARYSWXWOP5H6JXWJV2L677UEW2XOYINM6JVGRLYOR6GERBXS724Q, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:37.792348550 (60h49m47.514911591s) [noise=true , address=77.90.31.21:4888 , piece_id=HYKKKQFNZVE6YAXVY6EBPAAPMLANZ263V3IPM4FHOIIJRPWPVNKA, seg_pos={0 0}, upload=58329]: Uploading piece... Jul 27 00:25:38.889490612 (60h49m48.612053652s) [upload=58329]: Next returned nil segment Jul 27 00:25:38.889510590 (60h49m48.61207363s) [upload=58329]: Waiting for error group managing segments... Jul 27 00:25:39.175728011 (60h49m48.898291051s) [noise=true , address=79.127.205.235:4691 , piece_id=ARYSWXWOP5H6JXWJV2L677UEW2XOYINM6JVGRLYOR6GERBXS724Q, seg_pos={0 0}, upload=58329]: Done uploading piece. err: <nil> Jul 27 00:25:39.176514295 (60h49m48.899077335s) [noise=true , address=79.127.205.246:4283 , piece_id=754EA6V7C5DCY2NB7EAQDQHA46GOPHULCSTGYKD6K7APXZHB74PA, seg_pos={0 0}, upload=58329]: Done uploading piece. err: <nil> Jul 27 00:25:39.226390593 (60h49m48.948953633s) [noise=true , address=79.127.205.233:4633 , piece_id=5BUNLCRAAZWZKBDWZ5EU76GZHPAOH3LBM3DGGIPMZ5E5G7FZIEIA, seg_pos={0 0}, upload=58329]: Done uploading piece. err: <nil> Jul 27 00:25:39.239984332 (60h49m48.962547372s) [noise=true , address=64.49.13.140:4508 , piece_id=5US2CRXFGHOAZH7EX5ZZLF4WTCM5DVVBU6TUN424CUEMW7LFDB6A, seg_pos={0 0}, upload=58329]: Done uploading piece. err: <nil> Jul 27 00:25:39.621843945 (60h49m49.344406985s) [noise=true , address=79.127.205.245:4223 , piece_id=XIWL4D7CYS463HTW5R7WHWDQTSS4SQKEFR2X42DRV7KEBQULHEWQ, seg_pos={0 0}, upload=58329]: Done uploading piece. err: <nil> Jul 27 00:25:39.810524107 (60h49m49.533087147s) [noise=true , address=77.90.31.21:4888 , piece_id=HYKKKQFNZVE6YAXVY6EBPAAPMLANZ263V3IPM4FHOIIJRPWPVNKA, seg_pos={0 0}, upload=58329]: Done uploading piece. err: <nil> Jul 27 00:25:41.994844870 (60h49m51.71740791s) [noise=true , address=45.91.116.9:4277 , piece_id=RQHX7NWXBCY3K2A43MKYGLVSA5CM3IQ543GWTW2XVNNDBYGX3HJQ, seg_pos={0 0}, upload=58329]: Done uploading piece. err: <nil> Jul 27 00:25:42.201874922 (60h49m51.924437962s) [noise=true , address=79.127.220.101:4041 , piece_id=O5LA5NBW7BOQKOYQV6HRI6ZXGSRYDAT3MB6DUU2LZ5ZR2IJ32KJA, seg_pos={0 0}, upload=58329]: Done uploading piece. err: <nil> Jul 27 00:25:43.051712948 (60h49m52.774275988s) [noise=true , address=79.127.220.98:4422 , piece_id=N7TLVBUMEXIY5SZDPZG22GPBHERKCNUMKR5EHY6I34RDQWI5C3CQ, seg_pos={0 0}, upload=58329]: Done uploading piece. err: <nil> Jul 27 00:25:44.136271644 (60h49m53.858834684s) [noise=true , address=45.91.116.14:4194 , piece_id=XZHYYTFBGDOJQ6JZGMXYXWFGE7DJV7SH6XGNZSYT65OXCYOBEDHQ, seg_pos={0 0}, upload=58329]: Done uploading piece. err: <nil> Jul 27 00:25:45.290447420 (60h49m55.01301048s) [noise=true , address=79.127.201.215:4360 , piece_id=LTG5ARLINZPTM3AJOY7H46JD54HJ2AUX7G5IQJZK5RM4VMJFBUMQ, seg_pos={0 0}, upload=58329]: Done uploading piece. err: <nil> Jul 27 00:26:36.694800882 (60h50m46.417363922s) [noise=true , address=77.90.31.12:4600 , piece_id=NBAOBFIYWNQLY37FHRD4TCMRKB7GLRTAEPOK74CMRZKA2AXAQWZQ, seg_pos={0 0}, upload=58329]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:26:36.694947764 (60h50m46.417510804s) [noise=true , address=104.36.82.56:4230 , piece_id=IVAOPO5NSSH7RF634XKP3D2M4IGPQLDTX7EW3AFQ66TDGLBKHOZA, seg_pos={0 0}, upload=58329]: Done uploading piece. err: ecclient: upload failed (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, address:104.36.82.56:4230): protocol: write tcp 188.166.225.170:50078->104.36.82.56:4230: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:36.694990565 (60h50m46.417553595s) [noise=true , address=77.90.31.16:4760 , piece_id=O7OA65X4GPKEJSSNL6CMDPKHXAHE2VPGD7WRUTFFUYMXBAVOPACQ, seg_pos={0 0}, upload=58329]: Done uploading piece. err: ecclient: upload failed (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, address:77.90.31.16:4760): protocol: write tcp 188.166.225.170:51478->77.90.31.16:4760: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:36.695016144 (60h50m46.417579184s) [noise=true , address=104.36.82.11:4127 , piece_id=YPWJAI2ZB4APTMDPZTZL2IEKBMSWEX5NS4XP32Y6M2ZZYZCAIWFQ, seg_pos={0 0}, upload=58329]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: write tcp 188.166.225.170:37312->104.36.82.11:4127: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:36.695177834 (60h50m46.417740874s) [noise=true , address=104.36.82.46:4513 , piece_id=GIPIW45T3IXFFJ4GTIHLKGWD5WZPFAZEEF7VBFXX7FNZA7AMTUQQ, seg_pos={0 0}, upload=58329]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: write tcp 188.166.225.170:52174->104.36.82.46:4513: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:36.695229392 (60h50m46.417792432s) [noise=true , address=104.36.82.54:4174 , piece_id=OEMAKOQW433D24T7A3W42ZZXTT3FWOTWPDXQCQA4WLVJ4K6MFCJA, seg_pos={0 0}, upload=58329]: Done uploading piece. err: ecclient: upload failed (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51, address:104.36.82.54:4174): protocol: write tcp 188.166.225.170:40384->104.36.82.54:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:36.695281692 (60h50m46.417844732s) [noise=true , address=77.90.31.17:4793 , piece_id=O6SJMFD52DGD7Z5LDGMCAHVVM4CE45T4V37WMKC4COCQ7OVYJGWQ, seg_pos={0 0}, upload=58329]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: read tcp 188.166.225.170:51276->77.90.31.17:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:36.695335275 (60h50m46.417898325s) [seg_pos={0 0}, upload=58329]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:26:36.934612163 (60h50m46.657175203s) [upload=58329]: Done uploading segments... Jul 27 00:26:36.934634475 (60h50m46.657197515s) [upload=58329]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.241587087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX.] duration: 5.862940977s FAIL (2025-07-27 00:26:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.862965996s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [...X......X...XX..XX.X.XX...X...............XX......X...XXX.] duration: 41.323896257s FAIL (2025-07-27 00:26:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.323934289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............X..........................................X..] duration: 1.908757437s FAIL (2025-07-27 00:26:20 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.908778778s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXXX] duration: 30.238734285s FAIL (2025-07-27 00:26:10 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.238760544s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X.X....X.X....XXX.X...X......X.XX...XXXX.....X..........X.] duration: 7.083443207s FAIL (2025-07-27 00:25:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.083453888s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [....X......X...XX..XX.X.XX...X...............XX......X...XXX] duration: 1m0.001733751s FAIL (2025-07-27 00:25:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001752537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23187475s FAIL (2025-07-27 00:25:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 97s) [seg_pos={0 0}, upload=58315]: Done begin upload segment. Jul 27 00:24:37.334288710 (60h48m47.056851751s) [noise=true , address=77.90.31.18:4823 , piece_id=75YRBF3BLVSJAY4TS5K4SL23WU66LFNIGMAXT72VY5HP5JUAMHUQ, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.334307195 (60h48m47.056870235s) [noise=true , address=79.127.205.233:4643 , piece_id=7TXXKKPFCK2NOJVMSUFGHNJANWXSFQDL6DNSSWK2BB4W5YOQXSXA, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.334998738 (60h48m47.057561768s) [noise=true , address=104.36.82.46:4509 , piece_id=EGI6YQ7D5DZCRSYOWHMBQJ44B7ZO55I7USYAAJBRA4AZ7AWUPEMQ, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.335530216 (60h48m47.058093256s) [noise=true , address=104.36.82.12:4177 , piece_id=KEO7UTHV5NDQZMZBVUXEHLQNN2PVXG4S6BTAHEZEHVNOVLGFA7LA, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.335555815 (60h48m47.058118855s) [noise=true , address=79.127.205.234:4608 , piece_id=SJAIIQHHDV6VQMF5X3JZVNHVQFLLOIOUTR42EWQ5VJQQXNEKB2QQ, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.335969636 (60h48m47.058532687s) [noise=true , address=45.91.116.8:4062 , piece_id=ZJRGDD327WI4BOPKX7G5F7GAKIGHUNIITWFF5TGZ4RYJTUYTBWTQ, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.336030713 (60h48m47.058593763s) [noise=true , address=77.90.31.12:4615 , piece_id=RDPLRWF2BXRZAS6CV2LSORT2B6UBJNTQ6MZMA5PT7ZYOS6I4GOAQ, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.336335105 (60h48m47.058898155s) [noise=true , address=79.127.201.215:4346 , piece_id=3I3SATYKAX4K7DIQZ5TMYZXFK5Q345PUI6UZTDNN3HHOUA6KNT2Q, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.336425177 (60h48m47.058988218s) [noise=true , address=77.90.31.16:4748 , piece_id=RBZDHET4TF54A76JF53UKJBXFRDK3RSP5GL7EHWEXKXPWP6B52FA, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.336669966 (60h48m47.059233016s) [noise=true , address=77.90.31.11:4584 , piece_id=5WCJ6OLNF33HHLDERCEADIDQX7LY7HYN4GF3IRPDXHVLTRVYIZZQ, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.336773284 (60h48m47.059336324s) [noise=true , address=79.127.219.48:4074 , piece_id=ZAHQH5XY62MTJZDAYBJ5D3MMTUWMLMWMDA32W3BGBCXPWYVTPKFA, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.336996962 (60h48m47.059560002s) [noise=true , address=77.90.31.19:4875 , piece_id=NKQMS2ZMA3T5G5RZMXI5LAHKXULK5BLPTFKDRM2NRIYJ2MV7PO3Q, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.337294461 (60h48m47.059857491s) [noise=true , address=79.127.219.38:4004 , piece_id=KOSLKQ4RQMH4JYMKX5CXKFFFFOTSOYGSIAVUJK4NHH62U6EAP2CA, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.337598934 (60h48m47.060161974s) [noise=true , address=79.127.205.235:4678 , piece_id=Q2YH2ZQOT3C4YTTUYRWJVTPUPUXWXBXFMZAVLJSU3RXUT6KOXXRA, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.338009940 (60h48m47.06057298s) [noise=true , address=77.90.31.17:4792 , piece_id=7TFJFPN6MZT44LXORDGKY2T54ML6O2FPJP7E2DLCMJAHPLY2SZYA, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.338329662 (60h48m47.060892702s) [noise=true , address=104.36.82.47:4522 , piece_id=DPQ5V6I5QWPVB2725PTCTNBML746IO6KUSOIFD5V2BBTCSZO4TAQ, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.338445674 (60h48m47.061008724s) [noise=true , address=79.127.220.97:4361 , piece_id=5WHAGUQN6T73R3MZ5MXZ4Q5VMCPSUAZP7XM3QS6H4AOA6AAYUSNA, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:37.338755587 (60h48m47.061318627s) [noise=true , address=77.90.31.15:4722 , piece_id=Q56IQCWRU3NN5RGKZ5T554E5XGZRRGO6HLM3ND2K32SILMCJXZXQ, seg_pos={0 0}, upload=58315]: Uploading piece... Jul 27 00:24:38.428303890 (60h48m48.15086693s) [upload=58315]: Next returned nil segment Jul 27 00:24:38.428329990 (60h48m48.15089302s) [upload=58315]: Waiting for error group managing segments... Jul 27 00:24:38.714789152 (60h48m48.437352192s) [noise=true , address=79.127.205.233:4643 , piece_id=7TXXKKPFCK2NOJVMSUFGHNJANWXSFQDL6DNSSWK2BB4W5YOQXSXA, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:24:38.728061286 (60h48m48.450624316s) [noise=true , address=79.127.205.235:4678 , piece_id=Q2YH2ZQOT3C4YTTUYRWJVTPUPUXWXBXFMZAVLJSU3RXUT6KOXXRA, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:24:38.787440807 (60h48m48.510003847s) [noise=true , address=79.127.205.234:4608 , piece_id=SJAIIQHHDV6VQMF5X3JZVNHVQFLLOIOUTR42EWQ5VJQQXNEKB2QQ, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:24:38.881743626 (60h48m48.604306656s) [noise=true , address=79.127.219.48:4074 , piece_id=ZAHQH5XY62MTJZDAYBJ5D3MMTUWMLMWMDA32W3BGBCXPWYVTPKFA, seg_pos={0 0}, upload=58315]: Done uploading piece. err: ecclient: upload failed (node:127YpFQAmSFbgY2cmAEwcCPDUhg9AnRHS3SkcafEaQeSGVUwVdj, address:79.127.219.48:4074): protocol: not enough available disk space, have: 0, need: 6725632 Jul 27 00:24:39.295262803 (60h48m49.017825833s) [noise=true , address=79.127.219.38:4004 , piece_id=KOSLKQ4RQMH4JYMKX5CXKFFFFOTSOYGSIAVUJK4NHH62U6EAP2CA, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:24:39.321491867 (60h48m49.044054897s) [noise=true , address=77.90.31.18:4823 , piece_id=75YRBF3BLVSJAY4TS5K4SL23WU66LFNIGMAXT72VY5HP5JUAMHUQ, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:24:39.398696573 (60h48m49.121259613s) [noise=true , address=77.90.31.15:4722 , piece_id=Q56IQCWRU3NN5RGKZ5T554E5XGZRRGO6HLM3ND2K32SILMCJXZXQ, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:24:41.245237621 (60h48m50.967800661s) [noise=true , address=79.127.220.97:4361 , piece_id=5WHAGUQN6T73R3MZ5MXZ4Q5VMCPSUAZP7XM3QS6H4AOA6AAYUSNA, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:24:43.166335509 (60h48m52.888898549s) [noise=true , address=45.91.116.8:4062 , piece_id=ZJRGDD327WI4BOPKX7G5F7GAKIGHUNIITWFF5TGZ4RYJTUYTBWTQ, seg_pos={0 0}, upload=58315]: Done uploading piece. err: ecclient: upload failed (node:1ZgtJJK3pt2NqSfgFnXjZiYhDafhRdh6WkEf12AFTTuiku3Uhc, address:45.91.116.8:4062): protocol: storage node overloaded, request limit: 2000 Jul 27 00:24:47.603476894 (60h48m57.326039934s) [noise=true , address=77.90.31.19:4875 , piece_id=NKQMS2ZMA3T5G5RZMXI5LAHKXULK5BLPTFKDRM2NRIYJ2MV7PO3Q, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:25:02.743320214 (60h49m12.465883255s) [noise=true , address=79.127.201.215:4346 , piece_id=3I3SATYKAX4K7DIQZ5TMYZXFK5Q345PUI6UZTDNN3HHOUA6KNT2Q, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:25:23.706987212 (60h49m33.429550252s) [noise=true , address=77.90.31.12:4615 , piece_id=RDPLRWF2BXRZAS6CV2LSORT2B6UBJNTQ6MZMA5PT7ZYOS6I4GOAQ, seg_pos={0 0}, upload=58315]: Done uploading piece. err: <nil> Jul 27 00:25:36.225934902 (60h49m45.948497942s) [noise=true , address=77.90.31.16:4748 , piece_id=RBZDHET4TF54A76JF53UKJBXFRDK3RSP5GL7EHWEXKXPWP6B52FA, seg_pos={0 0}, upload=58315]: Done uploading piece. err: ecclient: upload failed (node:12k122BWfbKgBxSs8MCfNFkZYh2WfZDDHLktLQRiifkPr95G7SY, address:77.90.31.16:4748): protocol: rpc: dial tcp 77.90.31.16:4748: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:36.226054981 (60h49m45.948618021s) [noise=true , address=77.90.31.11:4584 , piece_id=5WCJ6OLNF33HHLDERCEADIDQX7LY7HYN4GF3IRPDXHVLTRVYIZZQ, seg_pos={0 0}, upload=58315]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:25:36.226500133 (60h49m45.949063173s) [noise=true , address=104.36.82.12:4177 , piece_id=KEO7UTHV5NDQZMZBVUXEHLQNN2PVXG4S6BTAHEZEHVNOVLGFA7LA, seg_pos={0 0}, upload=58315]: Done uploading piece. err: ecclient: upload failed (node:12rFuJ2JyBGWtaASJNnTdHoeKUY1mCfDYfJ74BkxK64U9it8FKF, address:104.36.82.12:4177): protocol: write tcp 188.166.225.170:44790->104.36.82.12:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:36.226655220 (60h49m45.94921827s) [noise=true , address=77.90.31.17:4792 , piece_id=7TFJFPN6MZT44LXORDGKY2T54ML6O2FPJP7E2DLCMJAHPLY2SZYA, seg_pos={0 0}, upload=58315]: Done uploading piece. err: ecclient: upload failed (node:12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN, address:77.90.31.17:4792): protocol: read tcp 188.166.225.170:40740->77.90.31.17:4792: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:36.226681340 (60h49m45.94924439s) [noise=true , address=104.36.82.47:4522 , piece_id=DPQ5V6I5QWPVB2725PTCTNBML746IO6KUSOIFD5V2BBTCSZO4TAQ, seg_pos={0 0}, upload=58315]: Done uploading piece. err: ecclient: upload failed (node:1zGZe6e37uop4y622fU1GbspMPVEZURS3FbMrf7je6VT9TYGFr, address:104.36.82.47:4522): protocol: rpc: dial tcp 104.36.82.47:4522: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:36.226701448 (60h49m45.949264488s) [noise=true , address=104.36.82.46:4509 , piece_id=EGI6YQ7D5DZCRSYOWHMBQJ44B7ZO55I7USYAAJBRA4AZ7AWUPEMQ, seg_pos={0 0}, upload=58315]: Done uploading piece. err: ecclient: upload failed (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh, address:104.36.82.46:4509): protocol: write tcp 188.166.225.170:38690->104.36.82.46:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:36.226733660 (60h49m45.9492967s) [seg_pos={0 0}, upload=58315]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:25:36.455809672 (60h49m46.178372702s) [upload=58315]: Done uploading segments... Jul 27 00:25:36.455833917 (60h49m46.178396957s) [upload=58315]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.231886733s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXXX] duration: 30.234363344s FAIL (2025-07-27 00:25:30 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.23440325s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X.] duration: 5.945013099s FAIL (2025-07-27 00:25:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 699 (60h48m30.885503729s) [upload=58310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:24:21.455742386 (60h48m31.178305426s) [upload=58310]: Done uploading segments... Jul 27 00:24:21.455761042 (60h48m31.178324082s) [upload=58310]: Upload finished. err: <nil> Jul 27 00:24:49.608066787 (60h48m59.330629827s) [upload=58318]: Starting upload Jul 27 00:24:49.608140568 (60h48m59.330703608s) [upload=58318]: Done starting upload Jul 27 00:24:49.608229108 (60h48m59.330792148s) [upload=58318]: Uploading segments... Jul 27 00:24:49.608362062 (60h48m59.330925102s) [upload=58318]: Got next segment. Inline: false Jul 27 00:24:49.608391879 (60h48m59.330954919s) [upload=58318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:24:49.608396478 (60h48m59.330959508s) [upload=58318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:24:49.844478273 (60h48m59.567041313s) [seg_pos={0 0}, upload=58318]: Begin upload segment... Jul 27 00:24:49.844522468 (60h48m59.567085498s) [seg_pos={0 0}, upload=58318]: Done begin upload segment. Jul 27 00:24:49.844555762 (60h48m59.567118792s) [noise=true , address=104.36.82.10:4078 , piece_id=5WOODLSDRRNLZNEPLLLCIQ5TDSBUGOMZFBALNYRYQ5367D5J5P3Q, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.844978750 (60h48m59.56754178s) [noise=true , address=45.91.116.6:4974 , piece_id=JIQ2CHQBKF6XDNBM67VHC33WEAQ4NHEMOO2VYRKKF6QQF4NCC37A, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.845300717 (60h48m59.567863757s) [noise=true , address=45.91.116.13:4154 , piece_id=DG3ARKYXOQ66SKBRRNNWOSKC4TZG2CRHX3HG3RMNKPSPSWAH6KQQ, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.845584860 (60h48m59.5681479s) [noise=true , address=77.90.31.14:4662 , piece_id=JBXY4ZOGAGF7DD66BXPA5GETIAAS7OEVKOJVFKGT7EWKESR66CGA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.845871880 (60h48m59.56843492s) [noise=true , address=77.90.31.13:4628 , piece_id=Q5XUIJPXGQJUTHMVDBOSUOSGCSIC457NW44RWOW57TIVIXXB4FVA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.846186351 (60h48m59.568749391s) [noise=true , address=45.91.116.5:4298 , piece_id=PETRENDPJ3D7OGSRZAZZVZKDFD2HJK2IMTYLDNID3VXWNPYXK3YQ, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.846458602 (60h48m59.569021642s) [noise=true , address=77.90.31.10:4520 , piece_id=27ZYCIE7Y53CJ26UVY6F2VGVXC24SK7DPCVZTZ2ZYSY7EZGF6Q2A, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.847020477 (60h48m59.569583527s) [noise=true , address=77.90.31.10:4518 , piece_id=TBJK4RAUN3ZWYDO2NTCM23ZTIKYD5GSVWXJQNGX4GMZBT6T55BZA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.847638309 (60h48m59.570201339s) [noise=true , address=77.90.31.16:4735 , piece_id=FCBHTIDEX2GYKR34DC6ZYEJBXP6BMZ7N7FLVLNXQO77KL5NOH37Q, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.847850616 (60h48m59.570413656s) [noise=true , address=45.91.116.11:4119 , piece_id=V2BH5AFKP66BLDE4NBI5FEOMCQWDPQ5FFV5UQS5DHV3DZUHSKDKA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848056950 (60h48m59.57061998s) [noise=true , address=45.91.116.8:4047 , piece_id=OVMDZISLDHJSM43PKQARKTDH6T5DGIE7JLQUAMUICGBV4PUXFHVA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848258606 (60h48m59.570821646s) [noise=true , address=45.91.116.11:4117 , piece_id=GSXCS2GX4TUVA4QHDINBTVBA64SUPZ3LVANLBBLE55Y3FT7HIGWA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848441055 (60h48m59.571004095s) [noise=true , address=104.36.82.3:4765 , piece_id=NQPXMYAOYTGPU3VSV5F6DZYUFLK2MMIPPLC636QBMWTOOOZOGNZA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848642931 (60h48m59.571205971s) [noise=true , address=45.91.116.4:4938 , piece_id=U54SMF27MEQTRL7FQYNR27DZAZE7M6MCM5K3YTU5KQI7O2I4Z33A, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848821513 (60h48m59.571384553s) [noise=true , address=104.36.82.8:4972 , piece_id=YVG5TAESB6XC4RD3YMUJTIL6TQRCO7LDAAE7M3XA7CIBYRUVWYNA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.849008110 (60h48m59.57157115s) [noise=true , address=104.36.82.11:4114 , piece_id=XBNSH7CVWG4LK2AL6IBAFFFERWAF3BVD7FZNRB5DSUAT5PMIS5PA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:51.509939290 (60h49m1.23250233s) [upload=58318]: Next returned nil segment Jul 27 00:24:51.509951304 (60h49m1.232514334s) [upload=58318]: Waiting for error group managing segments... Jul 27 00:24:51.848825802 (60h49m1.571388842s) [noise=true , address=45.91.116.13:4154 , piece_id=DG3ARKYXOQ66SKBRRNNWOSKC4TZG2CRHX3HG3RMNKPSPSWAH6KQQ, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.868516997 (60h49m1.591080037s) [noise=true , address=45.91.116.11:4117 , piece_id=GSXCS2GX4TUVA4QHDINBTVBA64SUPZ3LVANLBBLE55Y3FT7HIGWA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.870867164 (60h49m1.593430194s) [noise=true , address=45.91.116.4:4938 , piece_id=U54SMF27MEQTRL7FQYNR27DZAZE7M6MCM5K3YTU5KQI7O2I4Z33A, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.940969901 (60h49m1.663532981s) [noise=true , address=45.91.116.8:4047 , piece_id=OVMDZISLDHJSM43PKQARKTDH6T5DGIE7JLQUAMUICGBV4PUXFHVA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.948408754 (60h49m1.670971834s) [noise=true , address=45.91.116.6:4974 , piece_id=JIQ2CHQBKF6XDNBM67VHC33WEAQ4NHEMOO2VYRKKF6QQF4NCC37A, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.949260253 (60h49m1.671823293s) [noise=true , address=45.91.116.11:4119 , piece_id=V2BH5AFKP66BLDE4NBI5FEOMCQWDPQ5FFV5UQS5DHV3DZUHSKDKA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.958460255 (60h49m1.681023305s) [noise=true , address=45.91.116.5:4298 , piece_id=PETRENDPJ3D7OGSRZAZZVZKDFD2HJK2IMTYLDNID3VXWNPYXK3YQ, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.959566633 (60h49m1.682129673s) [noise=true , address=77.90.31.14:4662 , piece_id=JBXY4ZOGAGF7DD66BXPA5GETIAAS7OEVKOJVFKGT7EWKESR66CGA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:53.420609046 (60h49m3.143172086s) [noise=true , address=77.90.31.13:4628 , piece_id=Q5XUIJPXGQJUTHMVDBOSUOSGCSIC457NW44RWOW57TIVIXXB4FVA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:25:18.961139860 (60h49m28.6837029s) [noise=true , address=104.36.82.11:4114 , piece_id=XBNSH7CVWG4LK2AL6IBAFFFERWAF3BVD7FZNRB5DSUAT5PMIS5PA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961158235 (60h49m28.683721275s) [noise=true , address=104.36.82.8:4972 , piece_id=YVG5TAESB6XC4RD3YMUJTIL6TQRCO7LDAAE7M3XA7CIBYRUVWYNA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961196839 (60h49m28.683759879s) [noise=true , address=77.90.31.16:4735 , piece_id=FCBHTIDEX2GYKR34DC6ZYEJBXP6BMZ7N7FLVLNXQO77KL5NOH37Q, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961212729 (60h49m28.683775769s) [noise=true , address=77.90.31.10:4520 , piece_id=27ZYCIE7Y53CJ26UVY6F2VGVXC24SK7DPCVZTZ2ZYSY7EZGF6Q2A, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961239480 (60h49m28.68380252s) [noise=true , address=77.90.31.10:4518 , piece_id=TBJK4RAUN3ZWYDO2NTCM23ZTIKYD5GSVWXJQNGX4GMZBT6T55BZA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961252716 (60h49m28.683815746s) [noise=true , address=104.36.82.10:4078 , piece_id=5WOODLSDRRNLZNEPLLLCIQ5TDSBUGOMZFBALNYRYQ5367D5J5P3Q, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:58116->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961277724 (60h49m28.683840754s) [noise=true , address=104.36.82.3:4765 , piece_id=NQPXMYAOYTGPU3VSV5F6DZYUFLK2MMIPPLC636QBMWTOOOZOGNZA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961303412 (60h49m28.683866452s) [seg_pos={0 0}, upload=58318]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:25:19.188870085 (60h49m28.911433115s) [upload=58318]: Done uploading segments... Jul 27 00:25:19.188893971 (60h49m28.911457001s) [upload=58318]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration:* 5.945031744s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.X] duration: 30.229867622s FAIL (2025-07-27 00:25:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 699 (60h48m30.885503729s) [upload=58310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:24:21.455742386 (60h48m31.178305426s) [upload=58310]: Done uploading segments... Jul 27 00:24:21.455761042 (60h48m31.178324082s) [upload=58310]: Upload finished. err: <nil> Jul 27 00:24:49.608066787 (60h48m59.330629827s) [upload=58318]: Starting upload Jul 27 00:24:49.608140568 (60h48m59.330703608s) [upload=58318]: Done starting upload Jul 27 00:24:49.608229108 (60h48m59.330792148s) [upload=58318]: Uploading segments... Jul 27 00:24:49.608362062 (60h48m59.330925102s) [upload=58318]: Got next segment. Inline: false Jul 27 00:24:49.608391879 (60h48m59.330954919s) [upload=58318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:24:49.608396478 (60h48m59.330959508s) [upload=58318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:24:49.844478273 (60h48m59.567041313s) [seg_pos={0 0}, upload=58318]: Begin upload segment... Jul 27 00:24:49.844522468 (60h48m59.567085498s) [seg_pos={0 0}, upload=58318]: Done begin upload segment. Jul 27 00:24:49.844555762 (60h48m59.567118792s) [noise=true , address=104.36.82.10:4078 , piece_id=5WOODLSDRRNLZNEPLLLCIQ5TDSBUGOMZFBALNYRYQ5367D5J5P3Q, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.844978750 (60h48m59.56754178s) [noise=true , address=45.91.116.6:4974 , piece_id=JIQ2CHQBKF6XDNBM67VHC33WEAQ4NHEMOO2VYRKKF6QQF4NCC37A, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.845300717 (60h48m59.567863757s) [noise=true , address=45.91.116.13:4154 , piece_id=DG3ARKYXOQ66SKBRRNNWOSKC4TZG2CRHX3HG3RMNKPSPSWAH6KQQ, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.845584860 (60h48m59.5681479s) [noise=true , address=77.90.31.14:4662 , piece_id=JBXY4ZOGAGF7DD66BXPA5GETIAAS7OEVKOJVFKGT7EWKESR66CGA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.845871880 (60h48m59.56843492s) [noise=true , address=77.90.31.13:4628 , piece_id=Q5XUIJPXGQJUTHMVDBOSUOSGCSIC457NW44RWOW57TIVIXXB4FVA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.846186351 (60h48m59.568749391s) [noise=true , address=45.91.116.5:4298 , piece_id=PETRENDPJ3D7OGSRZAZZVZKDFD2HJK2IMTYLDNID3VXWNPYXK3YQ, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.846458602 (60h48m59.569021642s) [noise=true , address=77.90.31.10:4520 , piece_id=27ZYCIE7Y53CJ26UVY6F2VGVXC24SK7DPCVZTZ2ZYSY7EZGF6Q2A, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.847020477 (60h48m59.569583527s) [noise=true , address=77.90.31.10:4518 , piece_id=TBJK4RAUN3ZWYDO2NTCM23ZTIKYD5GSVWXJQNGX4GMZBT6T55BZA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.847638309 (60h48m59.570201339s) [noise=true , address=77.90.31.16:4735 , piece_id=FCBHTIDEX2GYKR34DC6ZYEJBXP6BMZ7N7FLVLNXQO77KL5NOH37Q, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.847850616 (60h48m59.570413656s) [noise=true , address=45.91.116.11:4119 , piece_id=V2BH5AFKP66BLDE4NBI5FEOMCQWDPQ5FFV5UQS5DHV3DZUHSKDKA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848056950 (60h48m59.57061998s) [noise=true , address=45.91.116.8:4047 , piece_id=OVMDZISLDHJSM43PKQARKTDH6T5DGIE7JLQUAMUICGBV4PUXFHVA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848258606 (60h48m59.570821646s) [noise=true , address=45.91.116.11:4117 , piece_id=GSXCS2GX4TUVA4QHDINBTVBA64SUPZ3LVANLBBLE55Y3FT7HIGWA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848441055 (60h48m59.571004095s) [noise=true , address=104.36.82.3:4765 , piece_id=NQPXMYAOYTGPU3VSV5F6DZYUFLK2MMIPPLC636QBMWTOOOZOGNZA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848642931 (60h48m59.571205971s) [noise=true , address=45.91.116.4:4938 , piece_id=U54SMF27MEQTRL7FQYNR27DZAZE7M6MCM5K3YTU5KQI7O2I4Z33A, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.848821513 (60h48m59.571384553s) [noise=true , address=104.36.82.8:4972 , piece_id=YVG5TAESB6XC4RD3YMUJTIL6TQRCO7LDAAE7M3XA7CIBYRUVWYNA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:49.849008110 (60h48m59.57157115s) [noise=true , address=104.36.82.11:4114 , piece_id=XBNSH7CVWG4LK2AL6IBAFFFERWAF3BVD7FZNRB5DSUAT5PMIS5PA, seg_pos={0 0}, upload=58318]: Uploading piece... Jul 27 00:24:51.509939290 (60h49m1.23250233s) [upload=58318]: Next returned nil segment Jul 27 00:24:51.509951304 (60h49m1.232514334s) [upload=58318]: Waiting for error group managing segments... Jul 27 00:24:51.848825802 (60h49m1.571388842s) [noise=true , address=45.91.116.13:4154 , piece_id=DG3ARKYXOQ66SKBRRNNWOSKC4TZG2CRHX3HG3RMNKPSPSWAH6KQQ, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.868516997 (60h49m1.591080037s) [noise=true , address=45.91.116.11:4117 , piece_id=GSXCS2GX4TUVA4QHDINBTVBA64SUPZ3LVANLBBLE55Y3FT7HIGWA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.870867164 (60h49m1.593430194s) [noise=true , address=45.91.116.4:4938 , piece_id=U54SMF27MEQTRL7FQYNR27DZAZE7M6MCM5K3YTU5KQI7O2I4Z33A, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.940969901 (60h49m1.663532981s) [noise=true , address=45.91.116.8:4047 , piece_id=OVMDZISLDHJSM43PKQARKTDH6T5DGIE7JLQUAMUICGBV4PUXFHVA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.948408754 (60h49m1.670971834s) [noise=true , address=45.91.116.6:4974 , piece_id=JIQ2CHQBKF6XDNBM67VHC33WEAQ4NHEMOO2VYRKKF6QQF4NCC37A, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.949260253 (60h49m1.671823293s) [noise=true , address=45.91.116.11:4119 , piece_id=V2BH5AFKP66BLDE4NBI5FEOMCQWDPQ5FFV5UQS5DHV3DZUHSKDKA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.958460255 (60h49m1.681023305s) [noise=true , address=45.91.116.5:4298 , piece_id=PETRENDPJ3D7OGSRZAZZVZKDFD2HJK2IMTYLDNID3VXWNPYXK3YQ, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:51.959566633 (60h49m1.682129673s) [noise=true , address=77.90.31.14:4662 , piece_id=JBXY4ZOGAGF7DD66BXPA5GETIAAS7OEVKOJVFKGT7EWKESR66CGA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:24:53.420609046 (60h49m3.143172086s) [noise=true , address=77.90.31.13:4628 , piece_id=Q5XUIJPXGQJUTHMVDBOSUOSGCSIC457NW44RWOW57TIVIXXB4FVA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: <nil> Jul 27 00:25:18.961139860 (60h49m28.6837029s) [noise=true , address=104.36.82.11:4114 , piece_id=XBNSH7CVWG4LK2AL6IBAFFFERWAF3BVD7FZNRB5DSUAT5PMIS5PA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961158235 (60h49m28.683721275s) [noise=true , address=104.36.82.8:4972 , piece_id=YVG5TAESB6XC4RD3YMUJTIL6TQRCO7LDAAE7M3XA7CIBYRUVWYNA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961196839 (60h49m28.683759879s) [noise=true , address=77.90.31.16:4735 , piece_id=FCBHTIDEX2GYKR34DC6ZYEJBXP6BMZ7N7FLVLNXQO77KL5NOH37Q, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961212729 (60h49m28.683775769s) [noise=true , address=77.90.31.10:4520 , piece_id=27ZYCIE7Y53CJ26UVY6F2VGVXC24SK7DPCVZTZ2ZYSY7EZGF6Q2A, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961239480 (60h49m28.68380252s) [noise=true , address=77.90.31.10:4518 , piece_id=TBJK4RAUN3ZWYDO2NTCM23ZTIKYD5GSVWXJQNGX4GMZBT6T55BZA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961252716 (60h49m28.683815746s) [noise=true , address=104.36.82.10:4078 , piece_id=5WOODLSDRRNLZNEPLLLCIQ5TDSBUGOMZFBALNYRYQ5367D5J5P3Q, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:58116->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961277724 (60h49m28.683840754s) [noise=true , address=104.36.82.3:4765 , piece_id=NQPXMYAOYTGPU3VSV5F6DZYUFLK2MMIPPLC636QBMWTOOOZOGNZA, seg_pos={0 0}, upload=58318]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:18.961303412 (60h49m28.683866452s) [seg_pos={0 0}, upload=58318]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:25:19.188870085 (60h49m28.911433115s) [upload=58318]: Done uploading segments... Jul 27 00:25:19.188893971 (60h49m28.911457001s) [upload=58318]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration:* 30.229886599s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X..........X] duration: 30.001082659s FAIL (2025-07-27 00:25:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00110952s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............X..........................................X.] duration: 1.866363284s FAIL (2025-07-27 00:25:05 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.86639745s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXXX] duration: 30.234269907s FAIL (2025-07-27 00:24:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.234289205s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.....X......X...XX..XX.X.XX...X...............XX......X...XX] duration: 1m0.001063366s FAIL (2025-07-27 00:24:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001075339s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.25533714s FAIL (2025-07-27 00:24:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== 4ITBHVQTFB2TPBOQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.837789598 (60h47m46.560352638s) [noise=true , address=79.127.219.47:4048 , piece_id=3C2ITTUR72D6BNWJHSACLD5FVSAWWMRCANC2YHYOHWSX6IFNWX5A, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.837922892 (60h47m46.560485932s) [noise=true , address=104.36.82.46:4508 , piece_id=ZKWZ66J253ERR6CBZZ3NBLX5TJUSUINWDKD7E2NBTPIUT2B5XWYQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.838073811 (60h47m46.560636841s) [noise=true , address=79.127.205.246:4284 , piece_id=EWBYPOYB57TF7ESYKFZDW3U6UGO7DAJFZHM7BBPY26F44N2BHYQA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.838326424 (60h47m46.560889464s) [noise=true , address=79.127.201.215:4358 , piece_id=QDVE6IR4ZANX7SXQPF7QUXBXGAPS2MCROSNHZEHTED55AGQD3BMQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.838561735 (60h47m46.561124775s) [noise=true , address=104.36.82.11:4130 , piece_id=EOUTV7ZFMCRBCR5ZNS7HMPNCKPKSBP6GTUOIKDPORMOBGA2BPHJA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.839328331 (60h47m46.561891371s) [noise=true , address=104.36.82.53:4131 , piece_id=M5BCO2IMSUTIFK6UWCNVXR5GPN6OQGI3JJDRMZNHFONM24AZBETA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.839588259 (60h47m46.562151299s) [noise=true , address=79.127.205.234:4619 , piece_id=6E56L4RSFUFBVNHAG7VOAPLDX7TPTSRAKZR2ZNTOIZ3H4XAS3BCA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.839865719 (60h47m46.562428759s) [noise=true , address=77.90.31.13:4659 , piece_id=CPK7G6RSRP5REL4XEKQFPDZK4NWA4SFL2H4LWKQGGGUM7FTQSMHA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.840168970 (60h47m46.56273201s) [noise=true , address=104.36.82.56:4230 , piece_id=W3XFY346SXAI5HGYCPPC74TJOEIDCTZXD2X55XDRYYTYB5WWNRGQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.840431182 (60h47m46.562994222s) [noise=true , address=45.91.116.14:4206 , piece_id=QNJ4IC7JYAJAOEH52YOTPCNU56O6UYJYUVKCP24NMSVGAYQHVDLQ, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.840459636 (60h47m46.563022676s) [noise=true , address=79.127.205.245:4242 , piece_id=MESWK4LX4HZLIYBNFMVWRLJ23EIYUQFZP3GHYVCFL5KZ4TA3MFBA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.840823823 (60h47m46.563386863s) [noise=true , address=104.36.82.52:4091 , piece_id=CDSJXFEQMOHFET66QNSSUZ3TTJGPJ4DL2QEBZIO5TMBCMQSFUQPA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.841195363 (60h47m46.563758403s) [noise=true , address=104.36.82.7:4918 , piece_id=4UP4UL5KEJB3WAR255LPD7QCLDDJFTJZUG4AYQHRHEPHTROQ3XMA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.841510827 (60h47m46.564073867s) [noise=true , address=64.49.13.138:4488 , piece_id=E4ZHOT6QY4L57Z7UBGEEDQSRDYUSMLVTFHOEVE4BMPOC5JEJR3NA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.841762979 (60h47m46.564326019s) [noise=true , address=79.127.205.235:4684 , piece_id=GNEQTGPWROAIRJARUD5T5PHG2M4BEMSNJFOEE3RYVMS3COYRD74Q, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:36.842054518 (60h47m46.564617568s) [noise=true , address=104.36.82.47:4532 , piece_id=AL35UOUZQF46QRL6LS5NRJPS6YETX3JWD3WL2GFM4CFLTHLY6NNA, seg_pos={0 0}, upload=58299]: Uploading piece... Jul 27 00:23:37.940047468 (60h47m47.662610508s) [upload=58299]: Next returned nil segment Jul 27 00:23:37.940062065 (60h47m47.662625105s) [upload=58299]: Waiting for error group managing segments... Jul 27 00:23:38.033467017 (60h47m47.756030057s) [noise=true , address=79.127.219.47:4048 , piece_id=3C2ITTUR72D6BNWJHSACLD5FVSAWWMRCANC2YHYOHWSX6IFNWX5A, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:23:38.249706559 (60h47m47.972269599s) [noise=true , address=79.127.205.245:4242 , piece_id=MESWK4LX4HZLIYBNFMVWRLJ23EIYUQFZP3GHYVCFL5KZ4TA3MFBA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:23:38.262287289 (60h47m47.984850329s) [noise=true , address=79.127.205.246:4284 , piece_id=EWBYPOYB57TF7ESYKFZDW3U6UGO7DAJFZHM7BBPY26F44N2BHYQA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:23:38.263857663 (60h47m47.986420704s) [noise=true , address=79.127.205.235:4684 , piece_id=GNEQTGPWROAIRJARUD5T5PHG2M4BEMSNJFOEE3RYVMS3COYRD74Q, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:23:38.300331980 (60h47m48.02289502s) [noise=true , address=79.127.205.234:4619 , piece_id=6E56L4RSFUFBVNHAG7VOAPLDX7TPTSRAKZR2ZNTOIZ3H4XAS3BCA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:23:39.264427374 (60h47m48.986990414s) [noise=true , address=45.91.116.14:4206 , piece_id=QNJ4IC7JYAJAOEH52YOTPCNU56O6UYJYUVKCP24NMSVGAYQHVDLQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:23:39.296307862 (60h47m49.018870902s) [noise=true , address=64.49.13.138:4488 , piece_id=E4ZHOT6QY4L57Z7UBGEEDQSRDYUSMLVTFHOEVE4BMPOC5JEJR3NA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:23:51.816611526 (60h48m1.539174666s) [noise=true , address=79.127.201.215:4358 , piece_id=QDVE6IR4ZANX7SXQPF7QUXBXGAPS2MCROSNHZEHTED55AGQD3BMQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:22.406894987 (60h48m32.129458027s) [noise=true , address=77.90.31.13:4659 , piece_id=CPK7G6RSRP5REL4XEKQFPDZK4NWA4SFL2H4LWKQGGGUM7FTQSMHA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: <nil> Jul 27 00:24:35.737981876 (60h48m45.460544906s) [noise=true , address=104.36.82.53:4131 , piece_id=M5BCO2IMSUTIFK6UWCNVXR5GPN6OQGI3JJDRMZNHFONM24AZBETA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:1uhyc5ozrF9ndeEnQXtjc9o3tsYiAnW1dzeuHV3kv1JS4T4a8C, address:104.36.82.53:4131): protocol: rpc: dial tcp 104.36.82.53:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:35.738027644 (60h48m45.460590684s) [noise=true , address=104.36.82.11:4130 , piece_id=EOUTV7ZFMCRBCR5ZNS7HMPNCKPKSBP6GTUOIKDPORMOBGA2BPHJA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12n69riZBquh3Qi436LVfVgNDJYuDxPzGi7d2BZDLurUJYVMCqs, address:104.36.82.11:4130): protocol: rpc: dial tcp 104.36.82.11:4130: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:35.738124469 (60h48m45.460687499s) [noise=true , address=104.36.82.52:4091 , piece_id=CDSJXFEQMOHFET66QNSSUZ3TTJGPJ4DL2QEBZIO5TMBCMQSFUQPA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:35.738159026 (60h48m45.460722066s) [noise=true , address=104.36.82.7:4918 , piece_id=4UP4UL5KEJB3WAR255LPD7QCLDDJFTJZUG4AYQHRHEPHTROQ3XMA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12EMGZ4hpXmBp4TwemLrbKAVtjBmtexMzEahRTzFw42qQipsAG, address:104.36.82.7:4918): protocol: write tcp 188.166.225.170:37276->104.36.82.7:4918: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:35.738190816 (60h48m45.460753856s) [noise=true , address=77.90.31.17:4774 , piece_id=55FG6E6IV64EKCLFYVYUCFKU24RRSVQXTAAC4ITBHVQTFB2TPBOQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: read tcp 188.166.225.170:45818->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:35.738195054 (60h48m45.460758094s) [noise=true , address=104.36.82.46:4508 , piece_id=ZKWZ66J253ERR6CBZZ3NBLX5TJUSUINWDKD7E2NBTPIUT2B5XWYQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:24:35.738239961 (60h48m45.460803001s) [noise=true , address=104.36.82.47:4532 , piece_id=AL35UOUZQF46QRL6LS5NRJPS6YETX3JWD3WL2GFM4CFLTHLY6NNA, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:24:35.738252795 (60h48m45.460815835s) [noise=true , address=104.36.82.56:4230 , piece_id=W3XFY346SXAI5HGYCPPC74TJOEIDCTZXD2X55XDRYYTYB5WWNRGQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, address:104.36.82.56:4230): protocol: rpc: dial tcp 104.36.82.56:4230: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:35.738283493 (60h48m45.460846533s) [noise=true , address=77.90.31.16:4747 , piece_id=UGSF7DBZUBBKNK7V5KAO5JR7ZMJAQMLHD6BR6WPLUG4UMZHWYHZQ, seg_pos={0 0}, upload=58299]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: rpc: dial tcp 77.90.31.16:4747: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:35.738298953 (60h48m45.460861993s) [seg_pos={0 0}, upload=58299]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:24:35.991568886 (60h48m45.714131926s) [upload=58299]: Done uploading segments... Jul 27 00:24:35.991593102 (60h48m45.714156142s) [upload=58299]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.255352441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXXX] duration: 30.242511869s FAIL (2025-07-27 00:24:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.24253325s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX.] duration: 7.116922434s FAIL (2025-07-27 00:24:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== FAHQFPRYQ2Q, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:49022->77.90.31.11:4555: use of closed network connection; conn closed: op Jul 27 00:23:25.766744612 (60h47m35.489307672s) [seg_pos={0 0}, upload=58291]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:23:25.999124114 (60h47m35.721687154s) [upload=58291]: Done uploading segments... Jul 27 00:23:25.999152989 (60h47m35.721716029s) [upload=58291]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:23:26.878833834 (60h47m36.601396864s) [upload=58298]: Starting upload Jul 27 00:23:26.878889149 (60h47m36.60145219s) [upload=58298]: Done starting upload Jul 27 00:23:26.878958201 (60h47m36.601521241s) [upload=58298]: Uploading segments... Jul 27 00:23:26.879205003 (60h47m36.601768043s) [upload=58298]: Got next segment. Inline: false Jul 27 00:23:26.879232696 (60h47m36.601795736s) [upload=58298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:23:26.879237807 (60h47m36.601800887s) [upload=58298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:23:27.117701870 (60h47m36.84026491s) [seg_pos={0 0}, upload=58298]: Begin upload segment... Jul 27 00:23:27.117772716 (60h47m36.840335756s) [seg_pos={0 0}, upload=58298]: Done begin upload segment. Jul 27 00:23:27.117840324 (60h47m36.840403364s) [noise=true , address=77.90.31.17:4771 , piece_id=GCNCJMYYVYM6DMCNWSAKKCTUFFD4EX3K3VJBR55HNA32X42G6PIQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.117860343 (60h47m36.840423383s) [noise=true , address=77.90.31.12:4589 , piece_id=6B4X7SBYKYWX36V3LIKQCVD4K2C6B73MC4FV4X7YTOFP3YDUJNSA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.118423991 (60h47m36.840987031s) [noise=true , address=45.91.116.9:4263 , piece_id=FXVIGA7V5RMTUNVM2JBJPQXAMY3M74NZBYI6PY3BCKDIULCWBFSA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.118698997 (60h47m36.841262037s) [noise=true , address=77.90.31.10:4520 , piece_id=WYRYFOMUQB2GWEKUSNOH5M47RM2TPU5Y3RASEJZOY6V4KN6HGC4Q, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119055539 (60h47m36.841618579s) [noise=true , address=104.36.82.6:4873 , piece_id=QMSIWKGGIRX6TP3I6ZS4VFWVGJKNJ3KCYK6ICD7SY2NIAH52TH6A, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119302452 (60h47m36.841865482s) [noise=true , address=45.91.116.12:4227 , piece_id=LGEUGQM2OKMZHSBS6JEPAJVU3MWZX32GJTY7N3JWMHMLP5A242CQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119503998 (60h47m36.842067028s) [noise=true , address=45.91.116.10:4081 , piece_id=JZ2BBF3YOO7LVVHQJRAC25QO7LLXG2QUEZ5OUV5B5LECGEAFQM5A, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119695213 (60h47m36.842258253s) [noise=true , address=45.91.116.7:4011 , piece_id=KYOR4ONTP5YTOCOQGR5LDEBHWQFDN7UUIPHYRYFUSJ6VCZ4QB7UQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119894365 (60h47m36.842457405s) [noise=true , address=45.91.116.11:4117 , piece_id=PBDXKCGNPEZNGAGH73LF64DQR4TXELLPRAFJFXNI4JIZENWGPIUA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120153350 (60h47m36.84271639s) [noise=true , address=45.91.116.4:4937 , piece_id=SKPNG3V7LDTOXQ56W3XQG4OMZBY6SQFA32MJDX262F3JRMASQKFQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120380224 (60h47m36.842943254s) [noise=true , address=77.90.31.14:4661 , piece_id=JICSMBTZ7ZBHY5I5SQEYYOGRCFRU4WI4L7YSEMDV7SH6LX5ZT4BQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120589725 (60h47m36.843152765s) [noise=true , address=77.90.31.11:4553 , piece_id=S6I22376H3SCXEFYYMGIG34LEFTLIRLZZDF4L7VQKM63FTXGALQA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120830175 (60h47m36.843393215s) [noise=true , address=45.91.116.8:4048 , piece_id=76J24RCKPFEW3HEOZZJ4EDTHNSVC2X2F3O4SUMKY4YU32SRKX66A, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.121036910 (60h47m36.84359995s) [noise=true , address=45.91.116.9:4262 , piece_id=56VHODDGIAEGUI666CZRZPPP76M4ITSWL3OQ7K5YGGPVQINWD2TA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.121246862 (60h47m36.843809892s) [noise=true , address=45.91.116.13:4153 , piece_id=Z324FERH3GMB2NVIEW4TCXK36LKDDFKY4NKH7FXFD35TRNNPIKSA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.121523792 (60h47m36.844086822s) [noise=true , address=77.90.31.10:4518 , piece_id=HO54YCT4JNGUP2VJBUQ6X5ZYHBVFLEPBWTETPGBDI7Q6HJZGTXTA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:28.854018534 (60h47m38.576581564s) [upload=58298]: Next returned nil segment Jul 27 00:23:28.854052358 (60h47m38.576615398s) [upload=58298]: Waiting for error group managing segments... Jul 27 00:23:29.129523117 (60h47m38.852086157s) [noise=true , address=45.91.116.12:4227 , piece_id=LGEUGQM2OKMZHSBS6JEPAJVU3MWZX32GJTY7N3JWMHMLP5A242CQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.155007436 (60h47m38.877570476s) [noise=true , address=45.91.116.9:4262 , piece_id=56VHODDGIAEGUI666CZRZPPP76M4ITSWL3OQ7K5YGGPVQINWD2TA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.155488357 (60h47m38.878051397s) [noise=true , address=45.91.116.10:4081 , piece_id=JZ2BBF3YOO7LVVHQJRAC25QO7LLXG2QUEZ5OUV5B5LECGEAFQM5A, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.157090652 (60h47m38.879653692s) [noise=true , address=45.91.116.9:4263 , piece_id=FXVIGA7V5RMTUNVM2JBJPQXAMY3M74NZBYI6PY3BCKDIULCWBFSA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.222674814 (60h47m38.945237854s) [noise=true , address=45.91.116.11:4117 , piece_id=PBDXKCGNPEZNGAGH73LF64DQR4TXELLPRAFJFXNI4JIZENWGPIUA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.223964532 (60h47m38.946527572s) [noise=true , address=45.91.116.4:4937 , piece_id=SKPNG3V7LDTOXQ56W3XQG4OMZBY6SQFA32MJDX262F3JRMASQKFQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.232334866 (60h47m38.954897907s) [noise=true , address=77.90.31.14:4661 , piece_id=JICSMBTZ7ZBHY5I5SQEYYOGRCFRU4WI4L7YSEMDV7SH6LX5ZT4BQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.236272311 (60h47m38.958835361s) [noise=true , address=45.91.116.13:4153 , piece_id=Z324FERH3GMB2NVIEW4TCXK36LKDDFKY4NKH7FXFD35TRNNPIKSA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.302653628 (60h47m39.025216708s) [noise=true , address=45.91.116.7:4011 , piece_id=KYOR4ONTP5YTOCOQGR5LDEBHWQFDN7UUIPHYRYFUSJ6VCZ4QB7UQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.356275302 (60h47m39.078838332s) [noise=true , address=45.91.116.8:4048 , piece_id=76J24RCKPFEW3HEOZZJ4EDTHNSVC2X2F3O4SUMKY4YU32SRKX66A, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:31.238097536 (60h47m40.960660576s) [noise=true , address=77.90.31.12:4589 , piece_id=6B4X7SBYKYWX36V3LIKQCVD4K2C6B73MC4FV4X7YTOFP3YDUJNSA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:56.226454714 (60h48m5.949017754s) [noise=true , address=77.90.31.10:4520 , piece_id=WYRYFOMUQB2GWEKUSNOH5M47RM2TPU5Y3RASEJZOY6V4KN6HGC4Q, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:51898->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226501363 (60h48m5.949064403s) [noise=true , address=77.90.31.10:4518 , piece_id=HO54YCT4JNGUP2VJBUQ6X5ZYHBVFLEPBWTETPGBDI7Q6HJZGTXTA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226622195 (60h48m5.949185235s) [noise=true , address=77.90.31.17:4771 , piece_id=GCNCJMYYVYM6DMCNWSAKKCTUFFD4EX3K3VJBR55HNA32X42G6PIQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39816->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226649257 (60h48m5.949212297s) [noise=true , address=104.36.82.6:4873 , piece_id=QMSIWKGGIRX6TP3I6ZS4VFWVGJKNJ3KCYK6ICD7SY2NIAH52TH6A, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226667200 (60h48m5.94923024s) [noise=true , address=77.90.31.11:4553 , piece_id=S6I22376H3SCXEFYYMGIG34LEFTLIRLZZDF4L7VQKM63FTXGALQA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: rpc: dial tcp 77.90.31.11:4553: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226683812 (60h48m5.949246852s) [seg_pos={0 0}, upload=58298]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:23:56.458682075 (60h48m6.181245115s) [upload=58298]: Done uploading segments... Jul 27 00:23:56.458702984 (60h48m6.181266025s) [upload=58298]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.116946749s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................X..........................................X] duration: 1.39121363s FAIL (2025-07-27 00:23:58 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.391241093s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXXX] duration: 30.233287166s FAIL (2025-07-27 00:23:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== FAHQFPRYQ2Q, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:49022->77.90.31.11:4555: use of closed network connection; conn closed: op Jul 27 00:23:25.766744612 (60h47m35.489307672s) [seg_pos={0 0}, upload=58291]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:23:25.999124114 (60h47m35.721687154s) [upload=58291]: Done uploading segments... Jul 27 00:23:25.999152989 (60h47m35.721716029s) [upload=58291]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:23:26.878833834 (60h47m36.601396864s) [upload=58298]: Starting upload Jul 27 00:23:26.878889149 (60h47m36.60145219s) [upload=58298]: Done starting upload Jul 27 00:23:26.878958201 (60h47m36.601521241s) [upload=58298]: Uploading segments... Jul 27 00:23:26.879205003 (60h47m36.601768043s) [upload=58298]: Got next segment. Inline: false Jul 27 00:23:26.879232696 (60h47m36.601795736s) [upload=58298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:23:26.879237807 (60h47m36.601800887s) [upload=58298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:23:27.117701870 (60h47m36.84026491s) [seg_pos={0 0}, upload=58298]: Begin upload segment... Jul 27 00:23:27.117772716 (60h47m36.840335756s) [seg_pos={0 0}, upload=58298]: Done begin upload segment. Jul 27 00:23:27.117840324 (60h47m36.840403364s) [noise=true , address=77.90.31.17:4771 , piece_id=GCNCJMYYVYM6DMCNWSAKKCTUFFD4EX3K3VJBR55HNA32X42G6PIQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.117860343 (60h47m36.840423383s) [noise=true , address=77.90.31.12:4589 , piece_id=6B4X7SBYKYWX36V3LIKQCVD4K2C6B73MC4FV4X7YTOFP3YDUJNSA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.118423991 (60h47m36.840987031s) [noise=true , address=45.91.116.9:4263 , piece_id=FXVIGA7V5RMTUNVM2JBJPQXAMY3M74NZBYI6PY3BCKDIULCWBFSA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.118698997 (60h47m36.841262037s) [noise=true , address=77.90.31.10:4520 , piece_id=WYRYFOMUQB2GWEKUSNOH5M47RM2TPU5Y3RASEJZOY6V4KN6HGC4Q, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119055539 (60h47m36.841618579s) [noise=true , address=104.36.82.6:4873 , piece_id=QMSIWKGGIRX6TP3I6ZS4VFWVGJKNJ3KCYK6ICD7SY2NIAH52TH6A, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119302452 (60h47m36.841865482s) [noise=true , address=45.91.116.12:4227 , piece_id=LGEUGQM2OKMZHSBS6JEPAJVU3MWZX32GJTY7N3JWMHMLP5A242CQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119503998 (60h47m36.842067028s) [noise=true , address=45.91.116.10:4081 , piece_id=JZ2BBF3YOO7LVVHQJRAC25QO7LLXG2QUEZ5OUV5B5LECGEAFQM5A, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119695213 (60h47m36.842258253s) [noise=true , address=45.91.116.7:4011 , piece_id=KYOR4ONTP5YTOCOQGR5LDEBHWQFDN7UUIPHYRYFUSJ6VCZ4QB7UQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.119894365 (60h47m36.842457405s) [noise=true , address=45.91.116.11:4117 , piece_id=PBDXKCGNPEZNGAGH73LF64DQR4TXELLPRAFJFXNI4JIZENWGPIUA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120153350 (60h47m36.84271639s) [noise=true , address=45.91.116.4:4937 , piece_id=SKPNG3V7LDTOXQ56W3XQG4OMZBY6SQFA32MJDX262F3JRMASQKFQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120380224 (60h47m36.842943254s) [noise=true , address=77.90.31.14:4661 , piece_id=JICSMBTZ7ZBHY5I5SQEYYOGRCFRU4WI4L7YSEMDV7SH6LX5ZT4BQ, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120589725 (60h47m36.843152765s) [noise=true , address=77.90.31.11:4553 , piece_id=S6I22376H3SCXEFYYMGIG34LEFTLIRLZZDF4L7VQKM63FTXGALQA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.120830175 (60h47m36.843393215s) [noise=true , address=45.91.116.8:4048 , piece_id=76J24RCKPFEW3HEOZZJ4EDTHNSVC2X2F3O4SUMKY4YU32SRKX66A, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.121036910 (60h47m36.84359995s) [noise=true , address=45.91.116.9:4262 , piece_id=56VHODDGIAEGUI666CZRZPPP76M4ITSWL3OQ7K5YGGPVQINWD2TA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.121246862 (60h47m36.843809892s) [noise=true , address=45.91.116.13:4153 , piece_id=Z324FERH3GMB2NVIEW4TCXK36LKDDFKY4NKH7FXFD35TRNNPIKSA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:27.121523792 (60h47m36.844086822s) [noise=true , address=77.90.31.10:4518 , piece_id=HO54YCT4JNGUP2VJBUQ6X5ZYHBVFLEPBWTETPGBDI7Q6HJZGTXTA, seg_pos={0 0}, upload=58298]: Uploading piece... Jul 27 00:23:28.854018534 (60h47m38.576581564s) [upload=58298]: Next returned nil segment Jul 27 00:23:28.854052358 (60h47m38.576615398s) [upload=58298]: Waiting for error group managing segments... Jul 27 00:23:29.129523117 (60h47m38.852086157s) [noise=true , address=45.91.116.12:4227 , piece_id=LGEUGQM2OKMZHSBS6JEPAJVU3MWZX32GJTY7N3JWMHMLP5A242CQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.155007436 (60h47m38.877570476s) [noise=true , address=45.91.116.9:4262 , piece_id=56VHODDGIAEGUI666CZRZPPP76M4ITSWL3OQ7K5YGGPVQINWD2TA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.155488357 (60h47m38.878051397s) [noise=true , address=45.91.116.10:4081 , piece_id=JZ2BBF3YOO7LVVHQJRAC25QO7LLXG2QUEZ5OUV5B5LECGEAFQM5A, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.157090652 (60h47m38.879653692s) [noise=true , address=45.91.116.9:4263 , piece_id=FXVIGA7V5RMTUNVM2JBJPQXAMY3M74NZBYI6PY3BCKDIULCWBFSA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.222674814 (60h47m38.945237854s) [noise=true , address=45.91.116.11:4117 , piece_id=PBDXKCGNPEZNGAGH73LF64DQR4TXELLPRAFJFXNI4JIZENWGPIUA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.223964532 (60h47m38.946527572s) [noise=true , address=45.91.116.4:4937 , piece_id=SKPNG3V7LDTOXQ56W3XQG4OMZBY6SQFA32MJDX262F3JRMASQKFQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.232334866 (60h47m38.954897907s) [noise=true , address=77.90.31.14:4661 , piece_id=JICSMBTZ7ZBHY5I5SQEYYOGRCFRU4WI4L7YSEMDV7SH6LX5ZT4BQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.236272311 (60h47m38.958835361s) [noise=true , address=45.91.116.13:4153 , piece_id=Z324FERH3GMB2NVIEW4TCXK36LKDDFKY4NKH7FXFD35TRNNPIKSA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.302653628 (60h47m39.025216708s) [noise=true , address=45.91.116.7:4011 , piece_id=KYOR4ONTP5YTOCOQGR5LDEBHWQFDN7UUIPHYRYFUSJ6VCZ4QB7UQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:29.356275302 (60h47m39.078838332s) [noise=true , address=45.91.116.8:4048 , piece_id=76J24RCKPFEW3HEOZZJ4EDTHNSVC2X2F3O4SUMKY4YU32SRKX66A, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:31.238097536 (60h47m40.960660576s) [noise=true , address=77.90.31.12:4589 , piece_id=6B4X7SBYKYWX36V3LIKQCVD4K2C6B73MC4FV4X7YTOFP3YDUJNSA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: <nil> Jul 27 00:23:56.226454714 (60h48m5.949017754s) [noise=true , address=77.90.31.10:4520 , piece_id=WYRYFOMUQB2GWEKUSNOH5M47RM2TPU5Y3RASEJZOY6V4KN6HGC4Q, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:51898->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226501363 (60h48m5.949064403s) [noise=true , address=77.90.31.10:4518 , piece_id=HO54YCT4JNGUP2VJBUQ6X5ZYHBVFLEPBWTETPGBDI7Q6HJZGTXTA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226622195 (60h48m5.949185235s) [noise=true , address=77.90.31.17:4771 , piece_id=GCNCJMYYVYM6DMCNWSAKKCTUFFD4EX3K3VJBR55HNA32X42G6PIQ, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39816->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226649257 (60h48m5.949212297s) [noise=true , address=104.36.82.6:4873 , piece_id=QMSIWKGGIRX6TP3I6ZS4VFWVGJKNJ3KCYK6ICD7SY2NIAH52TH6A, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226667200 (60h48m5.94923024s) [noise=true , address=77.90.31.11:4553 , piece_id=S6I22376H3SCXEFYYMGIG34LEFTLIRLZZDF4L7VQKM63FTXGALQA, seg_pos={0 0}, upload=58298]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: rpc: dial tcp 77.90.31.11:4553: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:56.226683812 (60h48m5.949246852s) [seg_pos={0 0}, upload=58298]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:23:56.458682075 (60h48m6.181245115s) [upload=58298]: Done uploading segments... Jul 27 00:23:56.458702984 (60h48m6.181266025s) [upload=58298]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.233300991s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X..........] duration: 8.186698978s FAIL (2025-07-27 00:23:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.186719747s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X.........] duration: 28.790575933s FAIL (2025-07-27 00:23:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.790597585s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.245761056s FAIL (2025-07-27 00:23:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== QOVEGNAHH7YHZ6JLWA, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.358889046 (60h46m46.081452086s) [noise=true , address=45.91.116.14:4206 , piece_id=UEG65X4ZHLXE5A4MVVO2NB32IIWFU6TYEZ6UM2HAN3LSZQ6P5AWQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.359306165 (60h46m46.081869215s) [noise=true , address=77.90.31.16:4736 , piece_id=OSQHUQ75WTKSKEVA4UW4PDPGYFB55562OQB6KHP7OE4GVKMKMR4A, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.359532648 (60h46m46.082095688s) [seg_pos={0 0}, upload=58286]: Done begin upload segment. Jul 27 00:22:36.359558287 (60h46m46.082121327s) [noise=true , address=79.127.205.233:4645 , piece_id=TA5YMHFL4P2PDSZJUMBENDS7JRUO5YA4XSWTHHF4MXZE4HYQSKIQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.360857503 (60h46m46.083420543s) [noise=true , address=104.36.82.52:4103 , piece_id=FESHL3LIBNB4ZN522RGNKAG65K5LTUCQ5G3DCWR5ESHBOSE4NA4Q, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.361150514 (60h46m46.083713554s) [noise=true , address=77.90.31.10:4535 , piece_id=2RLHLSENNQBPKCLPDZEOAXXL4HAVZEDKXOKC3XEYJBYG4OMZ2O2Q, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.361476588 (60h46m46.084039628s) [noise=true , address=77.90.31.11:4584 , piece_id=X6MXKANUP76HLLF2DVWB27TNQ5AFE2YBK7TMZE4ZGMKY5HGOMWJA, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.361767975 (60h46m46.084331015s) [noise=true , address=104.36.82.56:4234 , piece_id=HPU55E7EGQPQ2MO6EOVEQRTZ5PHB2EAC353CRPIBB3JD7JKAGKEA, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.361974490 (60h46m46.08453753s) [noise=true , address=79.127.205.245:4218 , piece_id=JVBNWJFETGQACITDTMN4AAPTK4DWKDHGKMRQYX3U2AXBDKZ3PIXQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.362101503 (60h46m46.084664543s) [noise=true , address=79.127.201.215:4350 , piece_id=CHEYZPZKM7VQCBOBENT6BSWJHYJPYUIYUF54NGBJ4L2EUEG3EACQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.362583535 (60h46m46.085146575s) [noise=true , address=79.127.219.38:4001 , piece_id=UE7IV4A3PL7MMWVIFWTE43AFC4EJT62QYZSLYODJQVBXNG3IXEQQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.362912635 (60h46m46.085475675s) [noise=true , address=104.36.82.47:4542 , piece_id=THLRASBMMBVBKR2DPLDFGS4V56U3K4PG6VBUSZDPRBRXSMO6FOTQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.362921011 (60h46m46.085484051s) [noise=true , address=79.127.205.235:4677 , piece_id=5RAX26ESH7U7BADWHVD254Y75CNKICNLZZCATNEEUUD3AWVXW2MQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.362956859 (60h46m46.085519899s) [noise=true , address=79.127.220.97:4361 , piece_id=NNVVZFQAVDSGTOXMMDN5CFX3432I26GZAR27BUYTO4RXQYSXFKYQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.363233869 (60h46m46.085796909s) [noise=true , address=77.90.31.18:4815 , piece_id=7MHGJR5HB2YKHPBKYXLV2SWE3FIORSX55DLQWBJZIC5Q3EE7TMVA, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.363246133 (60h46m46.085809173s) [noise=true , address=104.36.82.55:4225 , piece_id=EDEFOFIJEPIR77TKHGI6TFFDZD76577E2RBABYONANAE2ILK6YVQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.363537079 (60h46m46.086100119s) [noise=true , address=64.49.13.140:4508 , piece_id=SHX2AKPUOHLEKH2OY3TKMPQUIQEIAXFFMX4LZVGZT5TJ2BN7VK3A, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:36.363868323 (60h46m46.086431363s) [noise=true , address=79.127.220.101:4041 , piece_id=JIEZ6K5U2BRK2MIY6LFJH6OWWHX4ACZ4HIUQLFVLY7GOD67RXBNQ, seg_pos={0 0}, upload=58286]: Uploading piece... Jul 27 00:22:37.447665282 (60h46m47.170228332s) [upload=58286]: Next returned nil segment Jul 27 00:22:37.447692194 (60h46m47.170255234s) [upload=58286]: Waiting for error group managing segments... Jul 27 00:22:37.501411946 (60h46m47.223974986s) [noise=true , address=79.127.220.97:4361 , piece_id=NNVVZFQAVDSGTOXMMDN5CFX3432I26GZAR27BUYTO4RXQYSXFKYQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:22:37.745347919 (60h46m47.467910959s) [noise=true , address=79.127.205.234:4606 , piece_id=SYDVXXEWEHM25RAQEGWH47HFZZ7ZOHXDMNQOVEGNAHH7YHZ6JLWA, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:37.747484648 (60h46m47.470047688s) [noise=true , address=79.127.205.245:4218 , piece_id=JVBNWJFETGQACITDTMN4AAPTK4DWKDHGKMRQYX3U2AXBDKZ3PIXQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:37.781937336 (60h46m47.504500376s) [noise=true , address=79.127.205.233:4645 , piece_id=TA5YMHFL4P2PDSZJUMBENDS7JRUO5YA4XSWTHHF4MXZE4HYQSKIQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:38.030972264 (60h46m47.753535304s) [noise=true , address=79.127.205.235:4677 , piece_id=5RAX26ESH7U7BADWHVD254Y75CNKICNLZZCATNEEUUD3AWVXW2MQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:38.250917325 (60h46m47.973480365s) [noise=true , address=79.127.219.38:4001 , piece_id=UE7IV4A3PL7MMWVIFWTE43AFC4EJT62QYZSLYODJQVBXNG3IXEQQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:38.332626600 (60h46m48.05518964s) [noise=true , address=77.90.31.18:4815 , piece_id=7MHGJR5HB2YKHPBKYXLV2SWE3FIORSX55DLQWBJZIC5Q3EE7TMVA, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:40.304136871 (60h46m50.026699911s) [noise=true , address=79.127.220.101:4041 , piece_id=JIEZ6K5U2BRK2MIY6LFJH6OWWHX4ACZ4HIUQLFVLY7GOD67RXBNQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:41.704307309 (60h46m51.426870349s) [noise=true , address=64.49.13.140:4508 , piece_id=SHX2AKPUOHLEKH2OY3TKMPQUIQEIAXFFMX4LZVGZT5TJ2BN7VK3A, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:49.259512908 (60h46m58.982075948s) [noise=true , address=45.91.116.14:4206 , piece_id=UEG65X4ZHLXE5A4MVVO2NB32IIWFU6TYEZ6UM2HAN3LSZQ6P5AWQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:22:52.112812417 (60h47m1.835375457s) [noise=true , address=79.127.201.215:4350 , piece_id=CHEYZPZKM7VQCBOBENT6BSWJHYJPYUIYUF54NGBJ4L2EUEG3EACQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: <nil> Jul 27 00:23:35.254588588 (60h47m44.977151628s) [noise=true , address=77.90.31.16:4736 , piece_id=OSQHUQ75WTKSKEVA4UW4PDPGYFB55562OQB6KHP7OE4GVKMKMR4A, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, address:77.90.31.16:4736): protocol: rpc: dial tcp 77.90.31.16:4736: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.254765656 (60h47m44.977328686s) [noise=true , address=104.36.82.52:4103 , piece_id=FESHL3LIBNB4ZN522RGNKAG65K5LTUCQ5G3DCWR5ESHBOSE4NA4Q, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: rpc: dial tcp 104.36.82.52:4103: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.254823567 (60h47m44.977386607s) [noise=true , address=104.36.82.47:4542 , piece_id=THLRASBMMBVBKR2DPLDFGS4V56U3K4PG6VBUSZDPRBRXSMO6FOTQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:129FwaCA1FbzJbmRNzhQeVgygicJYHWkSS9fLnJbAG7KZPpokLU, address:104.36.82.47:4542): protocol: rpc: dial tcp 104.36.82.47:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.254862192 (60h47m44.977425222s) [noise=true , address=104.36.82.55:4225 , piece_id=EDEFOFIJEPIR77TKHGI6TFFDZD76577E2RBABYONANAE2ILK6YVQ, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, address:104.36.82.55:4225): protocol: write tcp 188.166.225.170:53120->104.36.82.55:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.254922136 (60h47m44.977485166s) [noise=true , address=104.36.82.56:4234 , piece_id=HPU55E7EGQPQ2MO6EOVEQRTZ5PHB2EAC353CRPIBB3JD7JKAGKEA, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:104.36.82.56:4234): protocol: rpc: dial tcp 104.36.82.56:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.254977973 (60h47m44.977541013s) [noise=true , address=77.90.31.10:4535 , piece_id=2RLHLSENNQBPKCLPDZEOAXXL4HAVZEDKXOKC3XEYJBYG4OMZ2O2Q, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: rpc: dial tcp 77.90.31.10:4535: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:23:35.255018531 (60h47m44.977581571s) [noise=true , address=77.90.31.11:4584 , piece_id=X6MXKANUP76HLLF2DVWB27TNQ5AFE2YBK7TMZE4ZGMKY5HGOMWJA, seg_pos={0 0}, upload=58286]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: write tcp 188.166.225.170:51642->77.90.31.11:4584: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:23:35.255039040 (60h47m44.97760207s) [seg_pos={0 0}, upload=58286]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:23:35.498418277 (60h47m45.220981307s) [upload=58286]: Done uploading segments... Jul 27 00:23:35.498445649 (60h47m45.221008689s) [upload=58286]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.245775625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [X.....X......X...XX..XX.X.XX...X...............XX......X...X] duration: 1m0.001072982s FAIL (2025-07-27 00:23:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001100766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X.XXXXXX] duration: 30.233196508s FAIL (2025-07-27 00:23:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.233223029s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....XX..XXXX] duration: 30.234025965s FAIL (2025-07-27 00:23:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== n closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:22:55.301966701 (60h47m5.024529751s) [seg_pos={0 0}, upload=58285]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:22:55.535080726 (60h47m5.257643776s) [upload=58285]: Done uploading segments... Jul 27 00:22:55.535111805 (60h47m5.257674845s) [upload=58285]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:22:56.419177168 (60h47m6.141740228s) [upload=58291]: Starting upload Jul 27 00:22:56.419245739 (60h47m6.141808779s) [upload=58291]: Done starting upload Jul 27 00:22:56.419303629 (60h47m6.14186667s) [upload=58291]: Uploading segments... Jul 27 00:22:56.419440371 (60h47m6.142003411s) [upload=58291]: Got next segment. Inline: false Jul 27 00:22:56.419485077 (60h47m6.142048117s) [upload=58291]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:22:56.419489145 (60h47m6.142052185s) [upload=58291]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:22:56.658833663 (60h47m6.381396703s) [seg_pos={0 0}, upload=58291]: Begin upload segment... Jul 27 00:22:56.658891664 (60h47m6.381454704s) [seg_pos={0 0}, upload=58291]: Done begin upload segment. Jul 27 00:22:56.658915349 (60h47m6.381478389s) [noise=true , address=77.90.31.16:4734 , piece_id=F7UPDL7MWZ4WNGMNIINOLBQKYQUYXE3MOXZ56NL3YBCI4CCFEXJA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.658926670 (60h47m6.381489721s) [noise=true , address=45.91.116.10:4083 , piece_id=ASGAJYDCPW5FG5KHPIC2P7XBXTR23ADNPKINBCEMF6UH4L3JCCSQ, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.658967740 (60h47m6.38153078s) [noise=true , address=45.91.116.12:4227 , piece_id=RBOHG2EVNANNCO4JKZDF54I5TLP2Z6VUE7CN2N7XZKD53P4KWV7A, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.659311056 (60h47m6.381874086s) [noise=true , address=77.90.31.11:4555 , piece_id=DU7BRWRRNAFGVAKVEV452BUBAQBDSMTNNH3BDJDDXFAHQFPRYQ2Q, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.659389747 (60h47m6.381952787s) [noise=true , address=104.36.82.3:4767 , piece_id=5C7TXKX3ZEOESJH5JHAT6FHYXLQR3MJ7A2GXG65N7ANUZFDBIMJA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.659789111 (60h47m6.382352151s) [noise=true , address=104.36.82.16:4294 , piece_id=JFUDEUHLS5HSWWHHNHKYGQPL5PGXMWQSDY5CJ33HTITJW3FW7CSA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.660189837 (60h47m6.382752877s) [noise=true , address=77.90.31.12:4589 , piece_id=KOGBKJ3T33RKFOZE64VNOGOFVQSQTCEY25P7VSCT53VOVS2QKIYQ, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.660536962 (60h47m6.383100012s) [noise=true , address=104.36.82.2:4730 , piece_id=QJFX6ATBI43Q5PISUZ3DYANVZLGJUJMXTQX5FHAOXOAJ3G4DEOMA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.660932729 (60h47m6.383495769s) [noise=true , address=45.91.116.14:4190 , piece_id=7SWEXNFQQV6P5TGO45ZRWIRBIKAGXXH2DXWI4YM4PSQ5QHSUZOHA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.661252481 (60h47m6.383815521s) [noise=true , address=104.36.82.15:4258 , piece_id=SIPLDNRTFDKWXVE5EP2JTVEDSY56RKALOUESGRZTKKIXK44CBEQA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.661491148 (60h47m6.384054188s) [noise=true , address=45.91.116.8:4045 , piece_id=OA5D77Z77FC5MFIZHS72ELP7ULW5YFL2GHBRRFK744KVFCAZCD4Q, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.661820648 (60h47m6.384383688s) [noise=true , address=77.90.31.15:4699 , piece_id=CVCTE2PSSO2G4AEOQ3QPZ6KEDXDVQGKVBWXJLTSXCFETF2YSPDEQ, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.662040749 (60h47m6.384603789s) [noise=true , address=45.91.116.8:4047 , piece_id=7JUJ2TWF5D2PIFI2HLK32EZ3S7PXC44P27FX3ZXUC7XQOWUN67QA, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.662292952 (60h47m6.384855992s) [noise=true , address=45.91.116.7:4009 , piece_id=3LZBGS5KMBZGKF2QVU763K5JCLNY5TFHPJ3WH2AC47OO4NQCUNHQ, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.662324492 (60h47m6.384887542s) [noise=true , address=45.91.116.6:4975 , piece_id=A5EARY6IPO5EK3ELDKXFX6WXPLEJ4LWQGWHXPGGX65K7O2IDQ32Q, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:56.662640596 (60h47m6.385203636s) [noise=true , address=45.91.116.5:4298 , piece_id=WCSH3WKWQPHN56HCCABOOWZ7664QM6RZYTGUNW362OEPQLFBSNYQ, seg_pos={0 0}, upload=58291]: Uploading piece... Jul 27 00:22:58.385865324 (60h47m8.108428364s) [upload=58291]: Next returned nil segment Jul 27 00:22:58.385883758 (60h47m8.108446798s) [upload=58291]: Waiting for error group managing segments... Jul 27 00:22:58.662217739 (60h47m8.384780779s) [noise=true , address=45.91.116.12:4227 , piece_id=RBOHG2EVNANNCO4JKZDF54I5TLP2Z6VUE7CN2N7XZKD53P4KWV7A, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.682691902 (60h47m8.405254942s) [noise=true , address=45.91.116.7:4009 , piece_id=3LZBGS5KMBZGKF2QVU763K5JCLNY5TFHPJ3WH2AC47OO4NQCUNHQ, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.683459410 (60h47m8.40602244s) [noise=true , address=45.91.116.8:4045 , piece_id=OA5D77Z77FC5MFIZHS72ELP7ULW5YFL2GHBRRFK744KVFCAZCD4Q, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.692085685 (60h47m8.414648725s) [noise=true , address=45.91.116.8:4047 , piece_id=7JUJ2TWF5D2PIFI2HLK32EZ3S7PXC44P27FX3ZXUC7XQOWUN67QA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.758650774 (60h47m8.481213814s) [noise=true , address=45.91.116.6:4975 , piece_id=A5EARY6IPO5EK3ELDKXFX6WXPLEJ4LWQGWHXPGGX65K7O2IDQ32Q, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.762657199 (60h47m8.485220239s) [noise=true , address=45.91.116.10:4083 , piece_id=ASGAJYDCPW5FG5KHPIC2P7XBXTR23ADNPKINBCEMF6UH4L3JCCSQ, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.829026684 (60h47m8.551589724s) [noise=true , address=45.91.116.14:4190 , piece_id=7SWEXNFQQV6P5TGO45ZRWIRBIKAGXXH2DXWI4YM4PSQ5QHSUZOHA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.855458056 (60h47m8.578021086s) [noise=true , address=45.91.116.5:4298 , piece_id=WCSH3WKWQPHN56HCCABOOWZ7664QM6RZYTGUNW362OEPQLFBSNYQ, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:22:58.878029681 (60h47m8.600592721s) [noise=true , address=77.90.31.15:4699 , piece_id=CVCTE2PSSO2G4AEOQ3QPZ6KEDXDVQGKVBWXJLTSXCFETF2YSPDEQ, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:23:00.182861607 (60h47m9.905424647s) [noise=true , address=77.90.31.12:4589 , piece_id=KOGBKJ3T33RKFOZE64VNOGOFVQSQTCEY25P7VSCT53VOVS2QKIYQ, seg_pos={0 0}, upload=58291]: Done uploading piece. err: <nil> Jul 27 00:23:25.766371949 (60h47m35.488934989s) [noise=true , address=104.36.82.2:4730 , piece_id=QJFX6ATBI43Q5PISUZ3DYANVZLGJUJMXTQX5FHAOXOAJ3G4DEOMA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:36976->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.766427906 (60h47m35.488990936s) [noise=true , address=104.36.82.15:4258 , piece_id=SIPLDNRTFDKWXVE5EP2JTVEDSY56RKALOUESGRZTKKIXK44CBEQA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.766545671 (60h47m35.489108711s) [noise=true , address=104.36.82.3:4767 , piece_id=5C7TXKX3ZEOESJH5JHAT6FHYXLQR3MJ7A2GXG65N7ANUZFDBIMJA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.766580197 (60h47m35.489143237s) [noise=true , address=77.90.31.16:4734 , piece_id=F7UPDL7MWZ4WNGMNIINOLBQKYQUYXE3MOXZ56NL3YBCI4CCFEXJA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:52150->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.766639831 (60h47m35.489202871s) [noise=true , address=104.36.82.16:4294 , piece_id=JFUDEUHLS5HSWWHHNHKYGQPL5PGXMWQSDY5CJ33HTITJW3FW7CSA, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 188.166.225.170:47750->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:25.766687352 (60h47m35.489250432s) [noise=true , address=77.90.31.11:4555 , piece_id=DU7BRWRRNAFGVAKVEV452BUBAQBDSMTNNH3BDJDDXFAHQFPRYQ2Q, seg_pos={0 0}, upload=58291]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:49022->77.90.31.11:4555: use of closed network connection; conn closed: op Jul 27 00:23:25.766744612 (60h47m35.489307672s) [seg_pos={0 0}, upload=58291]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:23:25.999124114 (60h47m35.721687154s) [upload=58291]: Done uploading segments... Jul 27 00:23:25.999152989 (60h47m35.721716029s) [upload=58291]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.234042968s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)