Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.XXX...X.......X...........X......X...X.X......X..X......X.] duration: 32.698064769s FAIL (2025-08-16 15:25:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:24:16.043180969 (189h58m10.474261101s) [upload=197652]: Starting upload Aug 16 15:24:16.043232859 (189h58m10.47431299s) [upload=197652]: Done starting upload Aug 16 15:24:16.043434676 (189h58m10.474514807s) [upload=197652]: Uploading segments... Aug 16 15:24:16.043547622 (189h58m10.474627753s) [upload=197652]: Got next segment. Inline: false Aug 16 15:24:16.044202655 (189h58m10.475282786s) [upload=197652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:24:16.044210340 (189h58m10.475290471s) [upload=197652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:24:33.584642687 (189h58m28.015722818s) [seg_pos={0 0}, upload=197652]: Begin upload segment... Aug 16 15:24:33.584685658 (189h58m28.01576579s) [seg_pos={0 0}, upload=197652]: Done begin upload segment. Aug 16 15:24:33.584702882 (189h58m28.015783013s) [noise=true , address=104.36.82.58:4280 , piece_id=D3D57RVMNPSCKBNC7CX5TE63CKSL5NAMSERK7LCGPTUMRFONWEUQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.584734041 (189h58m28.015814172s) [noise=true , address=104.36.82.5:4853 , piece_id=M64TNJ3A7UW2EA4DL5ELEGJHDTMRYDBZODIAP5KNRCN73TY46ABQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.585089772 (189h58m28.016169903s) [noise=true , address=77.90.31.15:4703 , piece_id=XJFI4EAVYQFEWLD3KG2CW5CFDGPIJFZAFZRVDBLLHG6DH2MSJ6YA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.585940379 (189h58m28.017020511s) [noise=true , address=45.91.116.5:4304 , piece_id=7MAXCT3O26GF2HCINQCQ2W7ETHDYMKJD35CPETK5PHIE4AUOJC2Q, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.586214754 (189h58m28.017294876s) [noise=true , address=104.36.82.60:4357 , piece_id=INZPQFVAVOJKFUL3QTFVR7FQ4XMSTQBGTTJJM6JYH7O4ZLKW2R4A, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.586499039 (189h58m28.01757916s) [noise=true , address=104.36.82.23:4449 , piece_id=NFA5Y4UHGWANC7MUM7PAEFFZ5U32G7CH2PZOTUH4LQMSKLCHR6UQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.586713138 (189h58m28.017793269s) [noise=true , address=79.127.220.101:4061 , piece_id=37BQH2XPRXYIKRQC5KNGEHMINGBHHNFBOSXNQHQEFV6VBDNTOE5A, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.586942437 (189h58m28.018022568s) [noise=true , address=77.90.31.19:4850 , piece_id=TPEUJP4O3ZSSCDMBDKNXOYAOOKEIBDSIG37J4RU247CNSB57QIPQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.587144163 (189h58m28.018224284s) [noise=true , address=104.36.82.17:4359 , piece_id=OEFJ66AHDUBTMXTUVB74WX2X5NJNQMZKXNDOJBBWXV2MNC24UGJA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.587349707 (189h58m28.018429838s) [noise=true , address=104.36.82.38:4691 , piece_id=TVVSYMAMJHWOXPFIXSIHC2NTXAJKKNNVRDVAHI62FEHZYSEREV2A, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.587546774 (189h58m28.018626895s) [noise=true , address=77.90.31.21:4900 , piece_id=DXFLRBHS2VOC7VXOO7MM4FDWT3D42WMP5MJV2CZI2HK62JKZEWQA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.588622011 (189h58m28.019702132s) [noise=true , address=45.91.116.8:4077 , piece_id=6EB5EURR4LUVA2SHP6FJ4BMKGEDIKRRW6UDLGXYLRYZZHYNIQNMA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.588854114 (189h58m28.019934245s) [noise=true , address=104.36.82.44:4877 , piece_id=JAUUYGIM2NTID6K32QUYVO3L7SH3E7P7ULRYB6BSU3OGAXTRXJUA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.589101488 (189h58m28.020181619s) [noise=true , address=77.90.31.11:4579 , piece_id=KBHEIYES4BEXEYTSRB3BXWFSO6HZSFVELQLTL4PXRWAYVI7GG2OQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.589320447 (189h58m28.020400568s) [noise=true , address=66.23.193.175:4558 , piece_id=5EZXYQN6V7VDXHIKZREGRHDGXVIJ5IRRY76W2PPYRIIMFPPCP6NQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.589534476 (189h58m28.020614597s) [noise=true , address=156.146.43.225:4457 , piece_id=TSUTEFPCTCCDE6FY5XZADRD4IPAC3QS3L23BLREVEASYIMGKYNKQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.589725933 (189h58m28.020806064s) [noise=true , address=77.90.31.12:4599 , piece_id=UDMWX2CVO4KPI7PWASJ5YD7BFQLK5GP7AGAU3ZACYUZG35ZY2CHA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.589853377 (189h58m28.020933508s) [noise=true , address=104.36.82.2:4751 , piece_id=64WVAIUFQQFTRG5F6KEELGXXCDO575TGPQCV2O4J3NIGC6MWULDA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:34.815342067 (189h58m29.246422198s) [upload=197652]: Next returned nil segment Aug 16 15:24:34.815359119 (189h58m29.24643924s) [upload=197652]: Waiting for error group managing segments... Aug 16 15:24:35.129613185 (189h58m29.560693316s) [noise=true , address=66.23.193.175:4558 , piece_id=5EZXYQN6V7VDXHIKZREGRHDGXVIJ5IRRY76W2PPYRIIMFPPCP6NQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:35.594697010 (189h58m30.025777141s) [noise=true , address=104.36.82.2:4751 , piece_id=64WVAIUFQQFTRG5F6KEELGXXCDO575TGPQCV2O4J3NIGC6MWULDA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:35.608397716 (189h58m30.039477837s) [noise=true , address=104.36.82.38:4691 , piece_id=TVVSYMAMJHWOXPFIXSIHC2NTXAJKKNNVRDVAHI62FEHZYSEREV2A, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:35.660219792 (189h58m30.091299923s) [noise=true , address=156.146.43.225:4457 , piece_id=TSUTEFPCTCCDE6FY5XZADRD4IPAC3QS3L23BLREVEASYIMGKYNKQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:39.049878441 (189h58m33.480958572s) [noise=true , address=104.36.82.17:4359 , piece_id=OEFJ66AHDUBTMXTUVB74WX2X5NJNQMZKXNDOJBBWXV2MNC24UGJA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:45.026661192 (189h58m39.457741323s) [noise=true , address=79.127.220.101:4061 , piece_id=37BQH2XPRXYIKRQC5KNGEHMINGBHHNFBOSXNQHQEFV6VBDNTOE5A, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:47.496042626 (189h58m41.927122757s) [noise=true , address=45.91.116.5:4304 , piece_id=7MAXCT3O26GF2HCINQCQ2W7ETHDYMKJD35CPETK5PHIE4AUOJC2Q, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:51.102194976 (189h58m45.533275107s) [noise=true , address=45.91.116.8:4077 , piece_id=6EB5EURR4LUVA2SHP6FJ4BMKGEDIKRRW6UDLGXYLRYZZHYNIQNMA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:52.685339387 (189h58m47.116419518s) [noise=true , address=77.90.31.19:4850 , piece_id=TPEUJP4O3ZSSCDMBDKNXOYAOOKEIBDSIG37J4RU247CNSB57QIPQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:56.496926897 (189h58m50.928007028s) [noise=true , address=77.90.31.11:4579 , piece_id=KBHEIYES4BEXEYTSRB3BXWFSO6HZSFVELQLTL4PXRWAYVI7GG2OQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:58.193475055 (189h58m52.624555186s) [noise=true , address=77.90.31.21:4900 , piece_id=DXFLRBHS2VOC7VXOO7MM4FDWT3D42WMP5MJV2CZI2HK62JKZEWQA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:25:05.414748015 (189h58m59.845828146s) [noise=true , address=77.90.31.12:4599 , piece_id=UDMWX2CVO4KPI7PWASJ5YD7BFQLK5GP7AGAU3ZACYUZG35ZY2CHA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:25:05.497516565 (189h58m59.928596686s) [noise=true , address=104.36.82.5:4853 , piece_id=M64TNJ3A7UW2EA4DL5ELEGJHDTMRYDBZODIAP5KNRCN73TY46ABQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:25:15.398949383 (189h59m9.830029514s) [noise=true , address=104.36.82.58:4280 , piece_id=D3D57RVMNPSCKBNC7CX5TE63CKSL5NAMSERK7LCGPTUMRFONWEUQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: ecclient: upload failed (node:1fgHaEFKK4CeGg4x3vywTuYfT3kmac31xqg2QpsahQNXmvjhW8, address:104.36.82.58:4280): protocol: rpc: dial tcp 104.36.82.58:4280: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:25:15.398978479 (189h59m9.83005861s) [noise=true , address=77.90.31.15:4703 , piece_id=XJFI4EAVYQFEWLD3KG2CW5CFDGPIJFZAFZRVDBLLHG6DH2MSJ6YA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:15.398989299 (189h59m9.83006943s) [noise=true , address=104.36.82.23:4449 , piece_id=NFA5Y4UHGWANC7MUM7PAEFFZ5U32G7CH2PZOTUH4LQMSKLCHR6UQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: ecclient: upload failed (node:12f14pHDEEamEnHmeZCe38rjaE6EYWYkGMPrZMsbSDmApN5rckq, address:104.36.82.23:4449): protocol: rpc: dial tcp 104.36.82.23:4449: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:25:15.399057961 (189h59m9.830138092s) [noise=true , address=104.36.82.60:4357 , piece_id=INZPQFVAVOJKFUL3QTFVR7FQ4XMSTQBGTTJJM6JYH7O4ZLKW2R4A, seg_pos={0 0}, upload=197652]: Done uploading piece. err: ecclient: upload failed (node:17EmBobdmk16xm9girzx3asYwkhdCKKZCCPrBLwWTS4ujMdGmi, address:104.36.82.60:4357): protocol: write tcp 188.166.225.170:33440->104.36.82.60:4357: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:25:15.399107014 (189h59m9.830187135s) [noise=true , address=104.36.82.44:4877 , piece_id=JAUUYGIM2NTID6K32QUYVO3L7SH3E7P7ULRYB6BSU3OGAXTRXJUA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: ecclient: upload failed (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH, address:104.36.82.44:4877): protocol: write tcp 188.166.225.170:38582->104.36.82.44:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:25:15.399125410 (189h59m9.830205541s) [seg_pos={0 0}, upload=197652]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:25:16.066732284 (189h59m10.497812425s) [upload=197652]: Done uploading segments... Aug 16 15:25:16.066761029 (189h59m10.49784117s) [upload=197652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 32.698087562s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 8.322329619s FAIL (2025-08-16 15:25:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:24:31.713535236 (189h58m26.144615367s) [upload=197656]: Starting upload Aug 16 15:24:31.713619227 (189h58m26.144699368s) [upload=197656]: Done starting upload Aug 16 15:24:31.713737553 (189h58m26.144817684s) [upload=197656]: Uploading segments... Aug 16 15:24:31.713927757 (189h58m26.145007888s) [upload=197656]: Got next segment. Inline: false Aug 16 15:24:31.713958767 (189h58m26.145038898s) [upload=197656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:24:31.713964087 (189h58m26.145044218s) [upload=197656]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:24:31.965966230 (189h58m26.397046361s) [seg_pos={0 0}, upload=197656]: Begin upload segment... Aug 16 15:24:31.966022287 (189h58m26.397102418s) [seg_pos={0 0}, upload=197656]: Done begin upload segment. Aug 16 15:24:31.966092512 (189h58m26.397172643s) [noise=true , address=104.36.82.15:4257 , piece_id=4ZHA63HVVQNOG7VOQPQPE5SMJUOEQJGBMGN3XKWOLKSKSICRZZ5A, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966098032 (189h58m26.397178163s) [noise=true , address=77.90.31.19:4843 , piece_id=SWLPBUBO5VN5Q4SLIR56EWSN23ZBJ4Z5ZEJHPGV7QTU6H6C55Y5A, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966484982 (189h58m26.397565123s) [noise=true , address=45.91.116.4:4937 , piece_id=PL77LU4EMDISVHJZS4277TR5335LX6K4M7OSECKW4OBZ36N2QHGA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966544245 (189h58m26.397624377s) [noise=true , address=104.36.82.13:4187 , piece_id=Y4TG6AZETBJI2OB5FSJADSD7VYMUAFP3IAUIZKSPOALJAXGK7QCQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966811587 (189h58m26.397891718s) [noise=true , address=104.36.82.10:4078 , piece_id=QTNVVS5CSZDXMH5MU2HE7TW3TUMEDW7MN4PVE6UC3SZL2Z5JAJGQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966841925 (189h58m26.397922056s) [noise=true , address=45.91.116.14:4189 , piece_id=BL6E654UFKTS4C47WRAHQPMX572WMY4INSRUNUJAPYM7HNAEJQHQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.967119035 (189h58m26.398199166s) [noise=true , address=45.91.116.12:4226 , piece_id=P5FBIIG2XRTZHPMSGLD3SITKRJP3OS5OIHN4RQOVQNOP6B3OWEHA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.967464687 (189h58m26.398544818s) [noise=true , address=45.91.116.5:4297 , piece_id=MB47C74K4S4TP42BDUNE63TARM4LAVOOBZZUYSKVWCFSQRDNNTTA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.969019572 (189h58m26.400099703s) [noise=true , address=104.36.82.7:4910 , piece_id=EBX5UXQKRZCKTPDQLPYYN7TIXXWZVBWC55Z5MF2KQGU2QKNO3V7Q, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.969522494 (189h58m26.400602626s) [noise=true , address=104.36.82.4:4803 , piece_id=6WI7HEL5WBDGEC3F7MP2XIRIMNVDZ5NH22KQ7EMY3Y2SJUU3WWXQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.970881715 (189h58m26.401961846s) [noise=true , address=104.36.82.11:4114 , piece_id=2JMPTAHQMGWJKWEJKWAR6FA2Q7QFZ5YZI6HEPWLBG6AVGMW7HAEQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.972633777 (189h58m26.403713908s) [noise=true , address=77.90.31.12:4589 , piece_id=RTR4C5UNUMB5BYL2EXNUQBVJXB5YXWSQ4BFKOBFOXJQNHINUHPYQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.976371920 (189h58m26.407452051s) [noise=true , address=45.91.116.12:4227 , piece_id=JZX727TKVWQXNDBZ2XC2ETVT7LUQHEFSJJUKYPUYLYFL7L4AHQOA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.979249175 (189h58m26.410329306s) [noise=true , address=45.91.116.8:4045 , piece_id=IG7ULTSFGHM5X3XBEITNQL2H7T4MTKFZTC7I56NTMER66TDFW5XQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.981856965 (189h58m26.412937096s) [noise=true , address=77.90.31.13:4627 , piece_id=SGC6CTVVBO4HCQABVWP6RRJQ47JUS74AUEMYGH4PJD253FHI2PTA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.982483704 (189h58m26.413563835s) [noise=true , address=104.36.82.15:4258 , piece_id=AZRDSFPTN2OG5R76WRUVHPXRBPDQY6FNRSL76HT7J76HXS2EQEPA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:33.738269197 (189h58m28.169349318s) [upload=197656]: Next returned nil segment Aug 16 15:24:33.738285217 (189h58m28.169365348s) [upload=197656]: Waiting for error group managing segments... Aug 16 15:24:34.030016178 (189h58m28.461096309s) [noise=true , address=104.36.82.4:4803 , piece_id=6WI7HEL5WBDGEC3F7MP2XIRIMNVDZ5NH22KQ7EMY3Y2SJUU3WWXQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.118877305 (189h58m28.549957477s) [noise=true , address=104.36.82.10:4078 , piece_id=QTNVVS5CSZDXMH5MU2HE7TW3TUMEDW7MN4PVE6UC3SZL2Z5JAJGQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.143486203 (189h58m28.574566324s) [noise=true , address=104.36.82.15:4258 , piece_id=AZRDSFPTN2OG5R76WRUVHPXRBPDQY6FNRSL76HT7J76HXS2EQEPA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.157539612 (189h58m28.588619743s) [noise=true , address=104.36.82.15:4257 , piece_id=4ZHA63HVVQNOG7VOQPQPE5SMJUOEQJGBMGN3XKWOLKSKSICRZZ5A, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.218374053 (189h58m28.649454184s) [noise=true , address=104.36.82.11:4114 , piece_id=2JMPTAHQMGWJKWEJKWAR6FA2Q7QFZ5YZI6HEPWLBG6AVGMW7HAEQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.255144550 (189h58m28.686224681s) [noise=true , address=104.36.82.7:4910 , piece_id=EBX5UXQKRZCKTPDQLPYYN7TIXXWZVBWC55Z5MF2KQGU2QKNO3V7Q, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:39.376370263 (189h58m33.807450394s) [noise=true , address=104.36.82.13:4187 , piece_id=Y4TG6AZETBJI2OB5FSJADSD7VYMUAFP3IAUIZKSPOALJAXGK7QCQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:53.711185057 (189h58m48.142265189s) [noise=true , address=77.90.31.12:4589 , piece_id=RTR4C5UNUMB5BYL2EXNUQBVJXB5YXWSQ4BFKOBFOXJQNHINUHPYQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:53.969589421 (189h58m48.400669552s) [noise=true , address=45.91.116.12:4226 , piece_id=P5FBIIG2XRTZHPMSGLD3SITKRJP3OS5OIHN4RQOVQNOP6B3OWEHA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:55.062531273 (189h58m49.493611404s) [noise=true , address=77.90.31.19:4843 , piece_id=SWLPBUBO5VN5Q4SLIR56EWSN23ZBJ4Z5ZEJHPGV7QTU6H6C55Y5A, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:55.538805139 (189h58m49.969885271s) [noise=true , address=45.91.116.14:4189 , piece_id=BL6E654UFKTS4C47WRAHQPMX572WMY4INSRUNUJAPYM7HNAEJQHQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:59.424656929 (189h58m53.85573705s) [noise=true , address=77.90.31.13:4627 , piece_id=SGC6CTVVBO4HCQABVWP6RRJQ47JUS74AUEMYGH4PJD253FHI2PTA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:25:01.053905605 (189h58m55.484985756s) [noise=true , address=45.91.116.8:4045 , piece_id=IG7ULTSFGHM5X3XBEITNQL2H7T4MTKFZTC7I56NTMER66TDFW5XQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:01.053968104 (189h58m55.485048276s) [noise=true , address=45.91.116.4:4937 , piece_id=PL77LU4EMDISVHJZS4277TR5335LX6K4M7OSECKW4OBZ36N2QHGA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:01.054098244 (189h58m55.485178375s) [noise=true , address=45.91.116.12:4227 , piece_id=JZX727TKVWQXNDBZ2XC2ETVT7LUQHEFSJJUKYPUYLYFL7L4AHQOA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:25:01.054113754 (189h58m55.485193915s) [noise=true , address=45.91.116.5:4297 , piece_id=MB47C74K4S4TP42BDUNE63TARM4LAVOOBZZUYSKVWCFSQRDNNTTA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection Aug 16 15:25:01.054139643 (189h58m55.485219825s) [seg_pos={0 0}, upload=197656]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:25:01.288581916 (189h58m55.719662047s) [upload=197656]: Done uploading segments... Aug 16 15:25:01.288609159 (189h58m55.71968929s) [upload=197656]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.322355099s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.XXX...X.......X...........X......X...X.X......X..X......X] duration: 1m0.6691248s FAIL (2025-08-16 15:25:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 16 15:24:16.043180969 (189h58m10.474261101s) [upload=197652]: Starting upload Aug 16 15:24:16.043232859 (189h58m10.47431299s) [upload=197652]: Done starting upload Aug 16 15:24:16.043434676 (189h58m10.474514807s) [upload=197652]: Uploading segments... Aug 16 15:24:16.043547622 (189h58m10.474627753s) [upload=197652]: Got next segment. Inline: false Aug 16 15:24:16.044202655 (189h58m10.475282786s) [upload=197652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:24:16.044210340 (189h58m10.475290471s) [upload=197652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:24:33.584642687 (189h58m28.015722818s) [seg_pos={0 0}, upload=197652]: Begin upload segment... Aug 16 15:24:33.584685658 (189h58m28.01576579s) [seg_pos={0 0}, upload=197652]: Done begin upload segment. Aug 16 15:24:33.584702882 (189h58m28.015783013s) [noise=true , address=104.36.82.58:4280 , piece_id=D3D57RVMNPSCKBNC7CX5TE63CKSL5NAMSERK7LCGPTUMRFONWEUQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.584734041 (189h58m28.015814172s) [noise=true , address=104.36.82.5:4853 , piece_id=M64TNJ3A7UW2EA4DL5ELEGJHDTMRYDBZODIAP5KNRCN73TY46ABQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.585089772 (189h58m28.016169903s) [noise=true , address=77.90.31.15:4703 , piece_id=XJFI4EAVYQFEWLD3KG2CW5CFDGPIJFZAFZRVDBLLHG6DH2MSJ6YA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.585940379 (189h58m28.017020511s) [noise=true , address=45.91.116.5:4304 , piece_id=7MAXCT3O26GF2HCINQCQ2W7ETHDYMKJD35CPETK5PHIE4AUOJC2Q, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.586214754 (189h58m28.017294876s) [noise=true , address=104.36.82.60:4357 , piece_id=INZPQFVAVOJKFUL3QTFVR7FQ4XMSTQBGTTJJM6JYH7O4ZLKW2R4A, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.586499039 (189h58m28.01757916s) [noise=true , address=104.36.82.23:4449 , piece_id=NFA5Y4UHGWANC7MUM7PAEFFZ5U32G7CH2PZOTUH4LQMSKLCHR6UQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.586713138 (189h58m28.017793269s) [noise=true , address=79.127.220.101:4061 , piece_id=37BQH2XPRXYIKRQC5KNGEHMINGBHHNFBOSXNQHQEFV6VBDNTOE5A, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.586942437 (189h58m28.018022568s) [noise=true , address=77.90.31.19:4850 , piece_id=TPEUJP4O3ZSSCDMBDKNXOYAOOKEIBDSIG37J4RU247CNSB57QIPQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.587144163 (189h58m28.018224284s) [noise=true , address=104.36.82.17:4359 , piece_id=OEFJ66AHDUBTMXTUVB74WX2X5NJNQMZKXNDOJBBWXV2MNC24UGJA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.587349707 (189h58m28.018429838s) [noise=true , address=104.36.82.38:4691 , piece_id=TVVSYMAMJHWOXPFIXSIHC2NTXAJKKNNVRDVAHI62FEHZYSEREV2A, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.587546774 (189h58m28.018626895s) [noise=true , address=77.90.31.21:4900 , piece_id=DXFLRBHS2VOC7VXOO7MM4FDWT3D42WMP5MJV2CZI2HK62JKZEWQA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.588622011 (189h58m28.019702132s) [noise=true , address=45.91.116.8:4077 , piece_id=6EB5EURR4LUVA2SHP6FJ4BMKGEDIKRRW6UDLGXYLRYZZHYNIQNMA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.588854114 (189h58m28.019934245s) [noise=true , address=104.36.82.44:4877 , piece_id=JAUUYGIM2NTID6K32QUYVO3L7SH3E7P7ULRYB6BSU3OGAXTRXJUA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.589101488 (189h58m28.020181619s) [noise=true , address=77.90.31.11:4579 , piece_id=KBHEIYES4BEXEYTSRB3BXWFSO6HZSFVELQLTL4PXRWAYVI7GG2OQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.589320447 (189h58m28.020400568s) [noise=true , address=66.23.193.175:4558 , piece_id=5EZXYQN6V7VDXHIKZREGRHDGXVIJ5IRRY76W2PPYRIIMFPPCP6NQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.589534476 (189h58m28.020614597s) [noise=true , address=156.146.43.225:4457 , piece_id=TSUTEFPCTCCDE6FY5XZADRD4IPAC3QS3L23BLREVEASYIMGKYNKQ, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.589725933 (189h58m28.020806064s) [noise=true , address=77.90.31.12:4599 , piece_id=UDMWX2CVO4KPI7PWASJ5YD7BFQLK5GP7AGAU3ZACYUZG35ZY2CHA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:33.589853377 (189h58m28.020933508s) [noise=true , address=104.36.82.2:4751 , piece_id=64WVAIUFQQFTRG5F6KEELGXXCDO575TGPQCV2O4J3NIGC6MWULDA, seg_pos={0 0}, upload=197652]: Uploading piece... Aug 16 15:24:34.815342067 (189h58m29.246422198s) [upload=197652]: Next returned nil segment Aug 16 15:24:34.815359119 (189h58m29.24643924s) [upload=197652]: Waiting for error group managing segments... Aug 16 15:24:35.129613185 (189h58m29.560693316s) [noise=true , address=66.23.193.175:4558 , piece_id=5EZXYQN6V7VDXHIKZREGRHDGXVIJ5IRRY76W2PPYRIIMFPPCP6NQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:35.594697010 (189h58m30.025777141s) [noise=true , address=104.36.82.2:4751 , piece_id=64WVAIUFQQFTRG5F6KEELGXXCDO575TGPQCV2O4J3NIGC6MWULDA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:35.608397716 (189h58m30.039477837s) [noise=true , address=104.36.82.38:4691 , piece_id=TVVSYMAMJHWOXPFIXSIHC2NTXAJKKNNVRDVAHI62FEHZYSEREV2A, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:35.660219792 (189h58m30.091299923s) [noise=true , address=156.146.43.225:4457 , piece_id=TSUTEFPCTCCDE6FY5XZADRD4IPAC3QS3L23BLREVEASYIMGKYNKQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:39.049878441 (189h58m33.480958572s) [noise=true , address=104.36.82.17:4359 , piece_id=OEFJ66AHDUBTMXTUVB74WX2X5NJNQMZKXNDOJBBWXV2MNC24UGJA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:45.026661192 (189h58m39.457741323s) [noise=true , address=79.127.220.101:4061 , piece_id=37BQH2XPRXYIKRQC5KNGEHMINGBHHNFBOSXNQHQEFV6VBDNTOE5A, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:47.496042626 (189h58m41.927122757s) [noise=true , address=45.91.116.5:4304 , piece_id=7MAXCT3O26GF2HCINQCQ2W7ETHDYMKJD35CPETK5PHIE4AUOJC2Q, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:51.102194976 (189h58m45.533275107s) [noise=true , address=45.91.116.8:4077 , piece_id=6EB5EURR4LUVA2SHP6FJ4BMKGEDIKRRW6UDLGXYLRYZZHYNIQNMA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:52.685339387 (189h58m47.116419518s) [noise=true , address=77.90.31.19:4850 , piece_id=TPEUJP4O3ZSSCDMBDKNXOYAOOKEIBDSIG37J4RU247CNSB57QIPQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:56.496926897 (189h58m50.928007028s) [noise=true , address=77.90.31.11:4579 , piece_id=KBHEIYES4BEXEYTSRB3BXWFSO6HZSFVELQLTL4PXRWAYVI7GG2OQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:24:58.193475055 (189h58m52.624555186s) [noise=true , address=77.90.31.21:4900 , piece_id=DXFLRBHS2VOC7VXOO7MM4FDWT3D42WMP5MJV2CZI2HK62JKZEWQA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:25:05.414748015 (189h58m59.845828146s) [noise=true , address=77.90.31.12:4599 , piece_id=UDMWX2CVO4KPI7PWASJ5YD7BFQLK5GP7AGAU3ZACYUZG35ZY2CHA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:25:05.497516565 (189h58m59.928596686s) [noise=true , address=104.36.82.5:4853 , piece_id=M64TNJ3A7UW2EA4DL5ELEGJHDTMRYDBZODIAP5KNRCN73TY46ABQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: <nil> Aug 16 15:25:15.398949383 (189h59m9.830029514s) [noise=true , address=104.36.82.58:4280 , piece_id=D3D57RVMNPSCKBNC7CX5TE63CKSL5NAMSERK7LCGPTUMRFONWEUQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: ecclient: upload failed (node:1fgHaEFKK4CeGg4x3vywTuYfT3kmac31xqg2QpsahQNXmvjhW8, address:104.36.82.58:4280): protocol: rpc: dial tcp 104.36.82.58:4280: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:25:15.398978479 (189h59m9.83005861s) [noise=true , address=77.90.31.15:4703 , piece_id=XJFI4EAVYQFEWLD3KG2CW5CFDGPIJFZAFZRVDBLLHG6DH2MSJ6YA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:15.398989299 (189h59m9.83006943s) [noise=true , address=104.36.82.23:4449 , piece_id=NFA5Y4UHGWANC7MUM7PAEFFZ5U32G7CH2PZOTUH4LQMSKLCHR6UQ, seg_pos={0 0}, upload=197652]: Done uploading piece. err: ecclient: upload failed (node:12f14pHDEEamEnHmeZCe38rjaE6EYWYkGMPrZMsbSDmApN5rckq, address:104.36.82.23:4449): protocol: rpc: dial tcp 104.36.82.23:4449: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:25:15.399057961 (189h59m9.830138092s) [noise=true , address=104.36.82.60:4357 , piece_id=INZPQFVAVOJKFUL3QTFVR7FQ4XMSTQBGTTJJM6JYH7O4ZLKW2R4A, seg_pos={0 0}, upload=197652]: Done uploading piece. err: ecclient: upload failed (node:17EmBobdmk16xm9girzx3asYwkhdCKKZCCPrBLwWTS4ujMdGmi, address:104.36.82.60:4357): protocol: write tcp 188.166.225.170:33440->104.36.82.60:4357: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:25:15.399107014 (189h59m9.830187135s) [noise=true , address=104.36.82.44:4877 , piece_id=JAUUYGIM2NTID6K32QUYVO3L7SH3E7P7ULRYB6BSU3OGAXTRXJUA, seg_pos={0 0}, upload=197652]: Done uploading piece. err: ecclient: upload failed (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH, address:104.36.82.44:4877): protocol: write tcp 188.166.225.170:38582->104.36.82.44:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:25:15.399125410 (189h59m9.830205541s) [seg_pos={0 0}, upload=197652]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:25:16.066732284 (189h59m10.497812425s) [upload=197652]: Done uploading segments... Aug 16 15:25:16.066761029 (189h59m10.49784117s) [upload=197652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.669154205s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 524.971504ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:25:14 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 524.998787ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [.....X.....X............X...X...X..................X.X..X...] duration: 7.377073028s FAIL (2025-08-16 15:25:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:1024 (size:1024) ======LOGS====== Aug 16 15:22:56.049719326 (189h56m50.480799457s) [upload=197629]: Starting upload Aug 16 15:22:56.049769061 (189h56m50.480849183s) [upload=197629]: Done starting upload Aug 16 15:22:56.049870887 (189h56m50.480951018s) [upload=197629]: Uploading segments... Aug 16 15:22:56.049943124 (189h56m50.481023255s) [upload=197629]: Got next segment. Inline: true Aug 16 15:22:56.049948064 (189h56m50.481028196s) [upload=197629]: Waiting for error group managing segments... Aug 16 15:22:56.049988942 (189h56m50.481069063s) [upload=197629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:22:56.049996076 (189h56m50.481076207s) [upload=197629]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:22:56.049998711 (189h56m50.481078842s) [upload=197629]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:23:06.836570219 (189h57m1.26765035s) [upload=197629]: Done uploading segments... Aug 16 15:23:06.836588485 (189h57m1.267668606s) [upload=197629]: Upload finished. err: <nil> Duration:* 7.377099769s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration: 6.690612931s FAIL (2025-08-16 15:25:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.690637897s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..........................................................X] duration: 30.236579733s FAIL (2025-08-16 15:25:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:24:31.713535236 (189h58m26.144615367s) [upload=197656]: Starting upload Aug 16 15:24:31.713619227 (189h58m26.144699368s) [upload=197656]: Done starting upload Aug 16 15:24:31.713737553 (189h58m26.144817684s) [upload=197656]: Uploading segments... Aug 16 15:24:31.713927757 (189h58m26.145007888s) [upload=197656]: Got next segment. Inline: false Aug 16 15:24:31.713958767 (189h58m26.145038898s) [upload=197656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:24:31.713964087 (189h58m26.145044218s) [upload=197656]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:24:31.965966230 (189h58m26.397046361s) [seg_pos={0 0}, upload=197656]: Begin upload segment... Aug 16 15:24:31.966022287 (189h58m26.397102418s) [seg_pos={0 0}, upload=197656]: Done begin upload segment. Aug 16 15:24:31.966092512 (189h58m26.397172643s) [noise=true , address=104.36.82.15:4257 , piece_id=4ZHA63HVVQNOG7VOQPQPE5SMJUOEQJGBMGN3XKWOLKSKSICRZZ5A, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966098032 (189h58m26.397178163s) [noise=true , address=77.90.31.19:4843 , piece_id=SWLPBUBO5VN5Q4SLIR56EWSN23ZBJ4Z5ZEJHPGV7QTU6H6C55Y5A, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966484982 (189h58m26.397565123s) [noise=true , address=45.91.116.4:4937 , piece_id=PL77LU4EMDISVHJZS4277TR5335LX6K4M7OSECKW4OBZ36N2QHGA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966544245 (189h58m26.397624377s) [noise=true , address=104.36.82.13:4187 , piece_id=Y4TG6AZETBJI2OB5FSJADSD7VYMUAFP3IAUIZKSPOALJAXGK7QCQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966811587 (189h58m26.397891718s) [noise=true , address=104.36.82.10:4078 , piece_id=QTNVVS5CSZDXMH5MU2HE7TW3TUMEDW7MN4PVE6UC3SZL2Z5JAJGQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.966841925 (189h58m26.397922056s) [noise=true , address=45.91.116.14:4189 , piece_id=BL6E654UFKTS4C47WRAHQPMX572WMY4INSRUNUJAPYM7HNAEJQHQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.967119035 (189h58m26.398199166s) [noise=true , address=45.91.116.12:4226 , piece_id=P5FBIIG2XRTZHPMSGLD3SITKRJP3OS5OIHN4RQOVQNOP6B3OWEHA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.967464687 (189h58m26.398544818s) [noise=true , address=45.91.116.5:4297 , piece_id=MB47C74K4S4TP42BDUNE63TARM4LAVOOBZZUYSKVWCFSQRDNNTTA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.969019572 (189h58m26.400099703s) [noise=true , address=104.36.82.7:4910 , piece_id=EBX5UXQKRZCKTPDQLPYYN7TIXXWZVBWC55Z5MF2KQGU2QKNO3V7Q, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.969522494 (189h58m26.400602626s) [noise=true , address=104.36.82.4:4803 , piece_id=6WI7HEL5WBDGEC3F7MP2XIRIMNVDZ5NH22KQ7EMY3Y2SJUU3WWXQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.970881715 (189h58m26.401961846s) [noise=true , address=104.36.82.11:4114 , piece_id=2JMPTAHQMGWJKWEJKWAR6FA2Q7QFZ5YZI6HEPWLBG6AVGMW7HAEQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.972633777 (189h58m26.403713908s) [noise=true , address=77.90.31.12:4589 , piece_id=RTR4C5UNUMB5BYL2EXNUQBVJXB5YXWSQ4BFKOBFOXJQNHINUHPYQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.976371920 (189h58m26.407452051s) [noise=true , address=45.91.116.12:4227 , piece_id=JZX727TKVWQXNDBZ2XC2ETVT7LUQHEFSJJUKYPUYLYFL7L4AHQOA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.979249175 (189h58m26.410329306s) [noise=true , address=45.91.116.8:4045 , piece_id=IG7ULTSFGHM5X3XBEITNQL2H7T4MTKFZTC7I56NTMER66TDFW5XQ, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.981856965 (189h58m26.412937096s) [noise=true , address=77.90.31.13:4627 , piece_id=SGC6CTVVBO4HCQABVWP6RRJQ47JUS74AUEMYGH4PJD253FHI2PTA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:31.982483704 (189h58m26.413563835s) [noise=true , address=104.36.82.15:4258 , piece_id=AZRDSFPTN2OG5R76WRUVHPXRBPDQY6FNRSL76HT7J76HXS2EQEPA, seg_pos={0 0}, upload=197656]: Uploading piece... Aug 16 15:24:33.738269197 (189h58m28.169349318s) [upload=197656]: Next returned nil segment Aug 16 15:24:33.738285217 (189h58m28.169365348s) [upload=197656]: Waiting for error group managing segments... Aug 16 15:24:34.030016178 (189h58m28.461096309s) [noise=true , address=104.36.82.4:4803 , piece_id=6WI7HEL5WBDGEC3F7MP2XIRIMNVDZ5NH22KQ7EMY3Y2SJUU3WWXQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.118877305 (189h58m28.549957477s) [noise=true , address=104.36.82.10:4078 , piece_id=QTNVVS5CSZDXMH5MU2HE7TW3TUMEDW7MN4PVE6UC3SZL2Z5JAJGQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.143486203 (189h58m28.574566324s) [noise=true , address=104.36.82.15:4258 , piece_id=AZRDSFPTN2OG5R76WRUVHPXRBPDQY6FNRSL76HT7J76HXS2EQEPA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.157539612 (189h58m28.588619743s) [noise=true , address=104.36.82.15:4257 , piece_id=4ZHA63HVVQNOG7VOQPQPE5SMJUOEQJGBMGN3XKWOLKSKSICRZZ5A, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.218374053 (189h58m28.649454184s) [noise=true , address=104.36.82.11:4114 , piece_id=2JMPTAHQMGWJKWEJKWAR6FA2Q7QFZ5YZI6HEPWLBG6AVGMW7HAEQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:34.255144550 (189h58m28.686224681s) [noise=true , address=104.36.82.7:4910 , piece_id=EBX5UXQKRZCKTPDQLPYYN7TIXXWZVBWC55Z5MF2KQGU2QKNO3V7Q, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:39.376370263 (189h58m33.807450394s) [noise=true , address=104.36.82.13:4187 , piece_id=Y4TG6AZETBJI2OB5FSJADSD7VYMUAFP3IAUIZKSPOALJAXGK7QCQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:53.711185057 (189h58m48.142265189s) [noise=true , address=77.90.31.12:4589 , piece_id=RTR4C5UNUMB5BYL2EXNUQBVJXB5YXWSQ4BFKOBFOXJQNHINUHPYQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:53.969589421 (189h58m48.400669552s) [noise=true , address=45.91.116.12:4226 , piece_id=P5FBIIG2XRTZHPMSGLD3SITKRJP3OS5OIHN4RQOVQNOP6B3OWEHA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:55.062531273 (189h58m49.493611404s) [noise=true , address=77.90.31.19:4843 , piece_id=SWLPBUBO5VN5Q4SLIR56EWSN23ZBJ4Z5ZEJHPGV7QTU6H6C55Y5A, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:55.538805139 (189h58m49.969885271s) [noise=true , address=45.91.116.14:4189 , piece_id=BL6E654UFKTS4C47WRAHQPMX572WMY4INSRUNUJAPYM7HNAEJQHQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:24:59.424656929 (189h58m53.85573705s) [noise=true , address=77.90.31.13:4627 , piece_id=SGC6CTVVBO4HCQABVWP6RRJQ47JUS74AUEMYGH4PJD253FHI2PTA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: <nil> Aug 16 15:25:01.053905605 (189h58m55.484985756s) [noise=true , address=45.91.116.8:4045 , piece_id=IG7ULTSFGHM5X3XBEITNQL2H7T4MTKFZTC7I56NTMER66TDFW5XQ, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:01.053968104 (189h58m55.485048276s) [noise=true , address=45.91.116.4:4937 , piece_id=PL77LU4EMDISVHJZS4277TR5335LX6K4M7OSECKW4OBZ36N2QHGA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:25:01.054098244 (189h58m55.485178375s) [noise=true , address=45.91.116.12:4227 , piece_id=JZX727TKVWQXNDBZ2XC2ETVT7LUQHEFSJJUKYPUYLYFL7L4AHQOA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:25:01.054113754 (189h58m55.485193915s) [noise=true , address=45.91.116.5:4297 , piece_id=MB47C74K4S4TP42BDUNE63TARM4LAVOOBZZUYSKVWCFSQRDNNTTA, seg_pos={0 0}, upload=197656]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40062->45.91.116.5:4297: use of closed network connection Aug 16 15:25:01.054139643 (189h58m55.485219825s) [seg_pos={0 0}, upload=197656]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:25:01.288581916 (189h58m55.719662047s) [upload=197656]: Done uploading segments... Aug 16 15:25:01.288609159 (189h58m55.71968929s) [upload=197656]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.236611915s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 753.977927ms FAIL (2025-08-16 15:24:53 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 754.011782ms edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 735.364412ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:24:46 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 735.395762ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [......X.....X............X...X...X..................X.X..X..] duration: 19.590234435s FAIL (2025-08-16 15:24:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:1024 (size:1024) ======LOGS====== Aug 16 15:22:56.049719326 (189h56m50.480799457s) [upload=197629]: Starting upload Aug 16 15:22:56.049769061 (189h56m50.480849183s) [upload=197629]: Done starting upload Aug 16 15:22:56.049870887 (189h56m50.480951018s) [upload=197629]: Uploading segments... Aug 16 15:22:56.049943124 (189h56m50.481023255s) [upload=197629]: Got next segment. Inline: true Aug 16 15:22:56.049948064 (189h56m50.481028196s) [upload=197629]: Waiting for error group managing segments... Aug 16 15:22:56.049988942 (189h56m50.481069063s) [upload=197629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:22:56.049996076 (189h56m50.481076207s) [upload=197629]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:22:56.049998711 (189h56m50.481078842s) [upload=197629]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:23:06.836570219 (189h57m1.26765035s) [upload=197629]: Done uploading segments... Aug 16 15:23:06.836588485 (189h57m1.267668606s) [upload=197629]: Upload finished. err: <nil> Duration:* 19.590264804s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 808.574122ms FAIL (2025-08-16 15:24:25 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 808.597186ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXXXXX......................................................] duration: 6.446194518s FAIL (2025-08-16 15:24:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.446217583s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 495.137539ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:24:16 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 495.175373ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.X.XXX...X.......X...........X......X...X.X......X..X......] duration: 19.168857418s FAIL (2025-08-16 15:24:03 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 16 15:19:01.935812548 (189h52m56.366892679s) [upload=197555]: Starting upload Aug 16 15:19:01.935878694 (189h52m56.366958825s) [upload=197555]: Done starting upload Aug 16 15:19:01.936004836 (189h52m56.367084977s) [upload=197555]: Uploading segments... Aug 16 15:19:01.936169772 (189h52m56.367249913s) [upload=197555]: Got next segment. Inline: false Aug 16 15:19:01.936208656 (189h52m56.367288787s) [upload=197555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:19:01.936215168 (189h52m56.367295299s) [upload=197555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:19:02.436838893 (189h52m56.867919024s) [seg_pos={0 0}, upload=197555]: Begin upload segment... Aug 16 15:19:02.436886765 (189h52m56.867966896s) [seg_pos={0 0}, upload=197555]: Done begin upload segment. Aug 16 15:19:02.436907163 (189h52m56.867987294s) [noise=true , address=156.146.43.225:4466 , piece_id=2OLSHINQEE4OFYXXGBBBJZ2KG56ZEF7DZBOYYDIDEP6KUA4LT4FA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.437289305 (189h52m56.868369436s) [noise=true , address=77.90.31.12:4610 , piece_id=ALZSIIYVTIWTJYHYLG54C6L74UJJS7J7E7KGWM7NJYIIHERGPSXA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.437558330 (189h52m56.868638461s) [noise=true , address=79.127.201.215:4358 , piece_id=NEI7SIHCJZCGWGRUII5D5BK6SZQYMAEUCELGPSSYTG3EIZTCIDEA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.437911165 (189h52m56.868991296s) [noise=true , address=104.36.82.60:4343 , piece_id=F5SNMIJCO6RSAHINP7E42YHIVPMADZRNYVLJ5QFKWLCOXBSBB2JQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.438034170 (189h52m56.869114311s) [noise=true , address=66.23.193.175:4558 , piece_id=RUSF3T2RC4BMOZY3Y6ABO53FTPERYR4HMBKZFLK7PO6IC74VULQQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.438305329 (189h52m56.86938546s) [noise=true , address=104.36.82.36:4647 , piece_id=UVDSHYQLMPNLQXDIPBY4UE54ECM32C23XB7Z3ZLCUZ7X7LNMMPKQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.438742846 (189h52m56.869822977s) [noise=true , address=104.36.82.44:4891 , piece_id=WUQIJ65KSB7JXUPPT76Q4RRBLS2M2URLQOHPUYOKMI5H7EHRP4CQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.439101793 (189h52m56.870181924s) [noise=true , address=104.36.82.23:4455 , piece_id=MUTC2CTG2JJXADBW5UFDWP3HPITXDLHDUV5O66Z33IEMED5LZX4A, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.439457373 (189h52m56.870537504s) [noise=true , address=77.90.31.21:4887 , piece_id=TDBVOK2C2KXN7GXIFTWRRHWFEVGZ4BL4D2UWRDDCSIN26V2DW7PA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.439874452 (189h52m56.870954583s) [noise=true , address=104.36.82.44:4892 , piece_id=HGAGP2XS6QC36CCDLY2BZYQ72ERTIZCY2DM3IUO244JKYPFNQEPQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.440189414 (189h52m56.871269545s) [noise=true , address=104.36.82.34:4554 , piece_id=5SHIHHITT2JZYE76IK2XJSWFXZ6MPCT64VUXOZRW5YAKXOOFPIZQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.440524916 (189h52m56.871605047s) [noise=true , address=45.91.116.14:4210 , piece_id=KDO7IN4W5XK4WYBUREGFLLPYKRFMTIVUM66XAXVMGLA2AEU6DBSA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.440668481 (189h52m56.871748612s) [noise=true , address=104.36.82.17:4360 , piece_id=RZG2LMV3LKWBPO4HROH4A5ZWIMA42HJ4BIDXCIQVS6V6HXEQC2LA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.440951482 (189h52m56.872031613s) [noise=true , address=45.91.116.11:4131 , piece_id=7AX2QA3CYUHIGB2CCSTO6ZG4CLTT75KXG443LZV2OLBH2AYMUI4A, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.441195680 (189h52m56.872275811s) [noise=true , address=104.36.82.5:4843 , piece_id=L4RFYJ6FNFFE4W2SXMZM5IPGC3A33BLNZL6YOGJP4H5J3PP23IJQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.441459664 (189h52m56.872539795s) [noise=true , address=77.90.31.10:4538 , piece_id=HBIQKO5NIYWLFME3HWAFV5ZHJ66UJ6WQCK3JVKF63UD7NT3W44UQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.441719812 (189h52m56.872799943s) [noise=true , address=104.36.82.59:4310 , piece_id=VFFOACQFSN4OQHCOQ6RF4J3IY2DWCSEWVK5HDCBTVMG4HLDZH7FA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.442070373 (189h52m56.873150504s) [noise=true , address=77.90.31.11:4560 , piece_id=6BRPAUQV6XJ6HG7XH33EHJGXE3VNP7ZARJVRZ2B5ISSNLVJJIWRQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:03.644194997 (189h52m58.075275128s) [upload=197555]: Next returned nil segment Aug 16 15:19:03.644210507 (189h52m58.075290638s) [upload=197555]: Waiting for error group managing segments... Aug 16 15:19:03.967420310 (189h52m58.398500431s) [noise=true , address=66.23.193.175:4558 , piece_id=RUSF3T2RC4BMOZY3Y6ABO53FTPERYR4HMBKZFLK7PO6IC74VULQQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:04.393219317 (189h52m58.824299448s) [noise=true , address=156.146.43.225:4466 , piece_id=2OLSHINQEE4OFYXXGBBBJZ2KG56ZEF7DZBOYYDIDEP6KUA4LT4FA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:04.420543528 (189h52m58.851623659s) [noise=true , address=79.127.201.215:4358 , piece_id=NEI7SIHCJZCGWGRUII5D5BK6SZQYMAEUCELGPSSYTG3EIZTCIDEA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:04.496410662 (189h52m58.927490793s) [noise=true , address=104.36.82.34:4554 , piece_id=5SHIHHITT2JZYE76IK2XJSWFXZ6MPCT64VUXOZRW5YAKXOOFPIZQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.013496194 (189h52m59.444576325s) [noise=true , address=104.36.82.17:4360 , piece_id=RZG2LMV3LKWBPO4HROH4A5ZWIMA42HJ4BIDXCIQVS6V6HXEQC2LA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.585176407 (189h53m0.016256528s) [noise=true , address=104.36.82.60:4343 , piece_id=F5SNMIJCO6RSAHINP7E42YHIVPMADZRNYVLJ5QFKWLCOXBSBB2JQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.696057916 (189h53m0.127138047s) [noise=true , address=45.91.116.14:4210 , piece_id=KDO7IN4W5XK4WYBUREGFLLPYKRFMTIVUM66XAXVMGLA2AEU6DBSA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.771410964 (189h53m0.202491096s) [noise=true , address=77.90.31.10:4538 , piece_id=HBIQKO5NIYWLFME3HWAFV5ZHJ66UJ6WQCK3JVKF63UD7NT3W44UQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.890871970 (189h53m0.321952101s) [noise=true , address=104.36.82.5:4843 , piece_id=L4RFYJ6FNFFE4W2SXMZM5IPGC3A33BLNZL6YOGJP4H5J3PP23IJQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:06.043290113 (189h53m0.474370244s) [noise=true , address=77.90.31.12:4610 , piece_id=ALZSIIYVTIWTJYHYLG54C6L74UJJS7J7E7KGWM7NJYIIHERGPSXA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:07.255416925 (189h53m1.686497056s) [noise=true , address=104.36.82.23:4455 , piece_id=MUTC2CTG2JJXADBW5UFDWP3HPITXDLHDUV5O66Z33IEMED5LZX4A, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:07.703672319 (189h53m2.13475245s) [noise=true , address=77.90.31.11:4560 , piece_id=6BRPAUQV6XJ6HG7XH33EHJGXE3VNP7ZARJVRZ2B5ISSNLVJJIWRQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:09.663182983 (189h53m4.094263154s) [noise=true , address=45.91.116.11:4131 , piece_id=7AX2QA3CYUHIGB2CCSTO6ZG4CLTT75KXG443LZV2OLBH2AYMUI4A, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:10.138250940 (189h53m4.569331071s) [noise=true , address=104.36.82.36:4647 , piece_id=UVDSHYQLMPNLQXDIPBY4UE54ECM32C23XB7Z3ZLCUZ7X7LNMMPKQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:10.138277671 (189h53m4.569357792s) [seg_pos={0 0}, upload=197555]: Segment reached optimal threshold of 14 pieces. Aug 16 15:19:10.138630746 (189h53m4.569710877s) [noise=true , address=77.90.31.21:4887 , piece_id=TDBVOK2C2KXN7GXIFTWRRHWFEVGZ4BL4D2UWRDDCSIN26V2DW7PA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ): protocol: write tcp 188.166.225.170:47620->77.90.31.21:4887: use of closed network connection; write tcp 188.166.225.170:47620->77.90.31.21:4887: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47620->77.90.31.21:4887: use of closed network connection Aug 16 15:19:10.138675432 (189h53m4.569755563s) [noise=true , address=104.36.82.59:4310 , piece_id=VFFOACQFSN4OQHCOQ6RF4J3IY2DWCSEWVK5HDCBTVMG4HLDZH7FA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX): protocol: rpc: dial tcp 104.36.82.59:4310: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:19:10.138686132 (189h53m4.569766263s) [noise=true , address=104.36.82.44:4892 , piece_id=HGAGP2XS6QC36CCDLY2BZYQ72ERTIZCY2DM3IUO244JKYPFNQEPQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5): protocol: rpc: dial tcp 104.36.82.44:4892: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:19:10.138706822 (189h53m4.569786953s) [noise=true , address=104.36.82.44:4891 , piece_id=WUQIJ65KSB7JXUPPT76Q4RRBLS2M2URLQOHPUYOKMI5H7EHRP4CQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: rpc: dial tcp 104.36.82.44:4891: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:19:10.138723964 (189h53m4.569804095s) [seg_pos={0 0}, upload=197555]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:19:10.138777527 (189h53m4.569857658s) [upload=197555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:19:10.138781004 (189h53m4.569861135s) [upload=197555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:19:15.602831811 (189h53m10.033911942s) [upload=197555]: Done uploading segments... Aug 16 15:19:15.602860435 (189h53m10.033940556s) [upload=197555]: Upload finished. err: <nil> Duration: 19.168885833s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.222817807s FAIL (2025-08-16 15:24:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.222840069s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [.XXX........................................................] duration: 784.885105ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:23:53 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 784.911806ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 15.118299552s FAIL (2025-08-16 15:23:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 15.118332055s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [.......X.....X............X...X...X..................X.X..X.] duration: 9.958165741s FAIL (2025-08-16 15:23:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:1024 (size:1024) ======LOGS====== Aug 16 15:22:56.049719326 (189h56m50.480799457s) [upload=197629]: Starting upload Aug 16 15:22:56.049769061 (189h56m50.480849183s) [upload=197629]: Done starting upload Aug 16 15:22:56.049870887 (189h56m50.480951018s) [upload=197629]: Uploading segments... Aug 16 15:22:56.049943124 (189h56m50.481023255s) [upload=197629]: Got next segment. Inline: true Aug 16 15:22:56.049948064 (189h56m50.481028196s) [upload=197629]: Waiting for error group managing segments... Aug 16 15:22:56.049988942 (189h56m50.481069063s) [upload=197629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:22:56.049996076 (189h56m50.481076207s) [upload=197629]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:22:56.049998711 (189h56m50.481078842s) [upload=197629]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:23:06.836570219 (189h57m1.26765035s) [upload=197629]: Done uploading segments... Aug 16 15:23:06.836588485 (189h57m1.267668606s) [upload=197629]: Upload finished. err: <nil> Duration:* 9.958193474s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 2.371213669s FAIL (2025-08-16 15:23:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.371231041s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 7.472149448s FAIL (2025-08-16 15:23:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.472172532s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.X.XXX...X.......X...........X......X...X.X......X..X.....] duration: 29.388735462s FAIL (2025-08-16 15:23:18 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 16 15:19:01.935812548 (189h52m56.366892679s) [upload=197555]: Starting upload Aug 16 15:19:01.935878694 (189h52m56.366958825s) [upload=197555]: Done starting upload Aug 16 15:19:01.936004836 (189h52m56.367084977s) [upload=197555]: Uploading segments... Aug 16 15:19:01.936169772 (189h52m56.367249913s) [upload=197555]: Got next segment. Inline: false Aug 16 15:19:01.936208656 (189h52m56.367288787s) [upload=197555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:19:01.936215168 (189h52m56.367295299s) [upload=197555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:19:02.436838893 (189h52m56.867919024s) [seg_pos={0 0}, upload=197555]: Begin upload segment... Aug 16 15:19:02.436886765 (189h52m56.867966896s) [seg_pos={0 0}, upload=197555]: Done begin upload segment. Aug 16 15:19:02.436907163 (189h52m56.867987294s) [noise=true , address=156.146.43.225:4466 , piece_id=2OLSHINQEE4OFYXXGBBBJZ2KG56ZEF7DZBOYYDIDEP6KUA4LT4FA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.437289305 (189h52m56.868369436s) [noise=true , address=77.90.31.12:4610 , piece_id=ALZSIIYVTIWTJYHYLG54C6L74UJJS7J7E7KGWM7NJYIIHERGPSXA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.437558330 (189h52m56.868638461s) [noise=true , address=79.127.201.215:4358 , piece_id=NEI7SIHCJZCGWGRUII5D5BK6SZQYMAEUCELGPSSYTG3EIZTCIDEA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.437911165 (189h52m56.868991296s) [noise=true , address=104.36.82.60:4343 , piece_id=F5SNMIJCO6RSAHINP7E42YHIVPMADZRNYVLJ5QFKWLCOXBSBB2JQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.438034170 (189h52m56.869114311s) [noise=true , address=66.23.193.175:4558 , piece_id=RUSF3T2RC4BMOZY3Y6ABO53FTPERYR4HMBKZFLK7PO6IC74VULQQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.438305329 (189h52m56.86938546s) [noise=true , address=104.36.82.36:4647 , piece_id=UVDSHYQLMPNLQXDIPBY4UE54ECM32C23XB7Z3ZLCUZ7X7LNMMPKQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.438742846 (189h52m56.869822977s) [noise=true , address=104.36.82.44:4891 , piece_id=WUQIJ65KSB7JXUPPT76Q4RRBLS2M2URLQOHPUYOKMI5H7EHRP4CQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.439101793 (189h52m56.870181924s) [noise=true , address=104.36.82.23:4455 , piece_id=MUTC2CTG2JJXADBW5UFDWP3HPITXDLHDUV5O66Z33IEMED5LZX4A, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.439457373 (189h52m56.870537504s) [noise=true , address=77.90.31.21:4887 , piece_id=TDBVOK2C2KXN7GXIFTWRRHWFEVGZ4BL4D2UWRDDCSIN26V2DW7PA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.439874452 (189h52m56.870954583s) [noise=true , address=104.36.82.44:4892 , piece_id=HGAGP2XS6QC36CCDLY2BZYQ72ERTIZCY2DM3IUO244JKYPFNQEPQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.440189414 (189h52m56.871269545s) [noise=true , address=104.36.82.34:4554 , piece_id=5SHIHHITT2JZYE76IK2XJSWFXZ6MPCT64VUXOZRW5YAKXOOFPIZQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.440524916 (189h52m56.871605047s) [noise=true , address=45.91.116.14:4210 , piece_id=KDO7IN4W5XK4WYBUREGFLLPYKRFMTIVUM66XAXVMGLA2AEU6DBSA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.440668481 (189h52m56.871748612s) [noise=true , address=104.36.82.17:4360 , piece_id=RZG2LMV3LKWBPO4HROH4A5ZWIMA42HJ4BIDXCIQVS6V6HXEQC2LA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.440951482 (189h52m56.872031613s) [noise=true , address=45.91.116.11:4131 , piece_id=7AX2QA3CYUHIGB2CCSTO6ZG4CLTT75KXG443LZV2OLBH2AYMUI4A, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.441195680 (189h52m56.872275811s) [noise=true , address=104.36.82.5:4843 , piece_id=L4RFYJ6FNFFE4W2SXMZM5IPGC3A33BLNZL6YOGJP4H5J3PP23IJQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.441459664 (189h52m56.872539795s) [noise=true , address=77.90.31.10:4538 , piece_id=HBIQKO5NIYWLFME3HWAFV5ZHJ66UJ6WQCK3JVKF63UD7NT3W44UQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.441719812 (189h52m56.872799943s) [noise=true , address=104.36.82.59:4310 , piece_id=VFFOACQFSN4OQHCOQ6RF4J3IY2DWCSEWVK5HDCBTVMG4HLDZH7FA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.442070373 (189h52m56.873150504s) [noise=true , address=77.90.31.11:4560 , piece_id=6BRPAUQV6XJ6HG7XH33EHJGXE3VNP7ZARJVRZ2B5ISSNLVJJIWRQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:03.644194997 (189h52m58.075275128s) [upload=197555]: Next returned nil segment Aug 16 15:19:03.644210507 (189h52m58.075290638s) [upload=197555]: Waiting for error group managing segments... Aug 16 15:19:03.967420310 (189h52m58.398500431s) [noise=true , address=66.23.193.175:4558 , piece_id=RUSF3T2RC4BMOZY3Y6ABO53FTPERYR4HMBKZFLK7PO6IC74VULQQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:04.393219317 (189h52m58.824299448s) [noise=true , address=156.146.43.225:4466 , piece_id=2OLSHINQEE4OFYXXGBBBJZ2KG56ZEF7DZBOYYDIDEP6KUA4LT4FA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:04.420543528 (189h52m58.851623659s) [noise=true , address=79.127.201.215:4358 , piece_id=NEI7SIHCJZCGWGRUII5D5BK6SZQYMAEUCELGPSSYTG3EIZTCIDEA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:04.496410662 (189h52m58.927490793s) [noise=true , address=104.36.82.34:4554 , piece_id=5SHIHHITT2JZYE76IK2XJSWFXZ6MPCT64VUXOZRW5YAKXOOFPIZQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.013496194 (189h52m59.444576325s) [noise=true , address=104.36.82.17:4360 , piece_id=RZG2LMV3LKWBPO4HROH4A5ZWIMA42HJ4BIDXCIQVS6V6HXEQC2LA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.585176407 (189h53m0.016256528s) [noise=true , address=104.36.82.60:4343 , piece_id=F5SNMIJCO6RSAHINP7E42YHIVPMADZRNYVLJ5QFKWLCOXBSBB2JQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.696057916 (189h53m0.127138047s) [noise=true , address=45.91.116.14:4210 , piece_id=KDO7IN4W5XK4WYBUREGFLLPYKRFMTIVUM66XAXVMGLA2AEU6DBSA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.771410964 (189h53m0.202491096s) [noise=true , address=77.90.31.10:4538 , piece_id=HBIQKO5NIYWLFME3HWAFV5ZHJ66UJ6WQCK3JVKF63UD7NT3W44UQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.890871970 (189h53m0.321952101s) [noise=true , address=104.36.82.5:4843 , piece_id=L4RFYJ6FNFFE4W2SXMZM5IPGC3A33BLNZL6YOGJP4H5J3PP23IJQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:06.043290113 (189h53m0.474370244s) [noise=true , address=77.90.31.12:4610 , piece_id=ALZSIIYVTIWTJYHYLG54C6L74UJJS7J7E7KGWM7NJYIIHERGPSXA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:07.255416925 (189h53m1.686497056s) [noise=true , address=104.36.82.23:4455 , piece_id=MUTC2CTG2JJXADBW5UFDWP3HPITXDLHDUV5O66Z33IEMED5LZX4A, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:07.703672319 (189h53m2.13475245s) [noise=true , address=77.90.31.11:4560 , piece_id=6BRPAUQV6XJ6HG7XH33EHJGXE3VNP7ZARJVRZ2B5ISSNLVJJIWRQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:09.663182983 (189h53m4.094263154s) [noise=true , address=45.91.116.11:4131 , piece_id=7AX2QA3CYUHIGB2CCSTO6ZG4CLTT75KXG443LZV2OLBH2AYMUI4A, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:10.138250940 (189h53m4.569331071s) [noise=true , address=104.36.82.36:4647 , piece_id=UVDSHYQLMPNLQXDIPBY4UE54ECM32C23XB7Z3ZLCUZ7X7LNMMPKQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:10.138277671 (189h53m4.569357792s) [seg_pos={0 0}, upload=197555]: Segment reached optimal threshold of 14 pieces. Aug 16 15:19:10.138630746 (189h53m4.569710877s) [noise=true , address=77.90.31.21:4887 , piece_id=TDBVOK2C2KXN7GXIFTWRRHWFEVGZ4BL4D2UWRDDCSIN26V2DW7PA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ): protocol: write tcp 188.166.225.170:47620->77.90.31.21:4887: use of closed network connection; write tcp 188.166.225.170:47620->77.90.31.21:4887: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47620->77.90.31.21:4887: use of closed network connection Aug 16 15:19:10.138675432 (189h53m4.569755563s) [noise=true , address=104.36.82.59:4310 , piece_id=VFFOACQFSN4OQHCOQ6RF4J3IY2DWCSEWVK5HDCBTVMG4HLDZH7FA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX): protocol: rpc: dial tcp 104.36.82.59:4310: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:19:10.138686132 (189h53m4.569766263s) [noise=true , address=104.36.82.44:4892 , piece_id=HGAGP2XS6QC36CCDLY2BZYQ72ERTIZCY2DM3IUO244JKYPFNQEPQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5): protocol: rpc: dial tcp 104.36.82.44:4892: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:19:10.138706822 (189h53m4.569786953s) [noise=true , address=104.36.82.44:4891 , piece_id=WUQIJ65KSB7JXUPPT76Q4RRBLS2M2URLQOHPUYOKMI5H7EHRP4CQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: rpc: dial tcp 104.36.82.44:4891: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:19:10.138723964 (189h53m4.569804095s) [seg_pos={0 0}, upload=197555]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:19:10.138777527 (189h53m4.569857658s) [upload=197555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:19:10.138781004 (189h53m4.569861135s) [upload=197555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:19:15.602831811 (189h53m10.033911942s) [upload=197555]: Done uploading segments... Aug 16 15:19:15.602860435 (189h53m10.033940556s) [upload=197555]: Upload finished. err: <nil> Duration: 29.388757313s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 467.403844ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:23:17 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 467.434482ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [........X.....X............X...X...X..................X.X..X] duration: 21.401579356s FAIL (2025-08-16 15:23:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:1024 (size:1024) ======LOGS====== Aug 16 15:22:56.049719326 (189h56m50.480799457s) [upload=197629]: Starting upload Aug 16 15:22:56.049769061 (189h56m50.480849183s) [upload=197629]: Done starting upload Aug 16 15:22:56.049870887 (189h56m50.480951018s) [upload=197629]: Uploading segments... Aug 16 15:22:56.049943124 (189h56m50.481023255s) [upload=197629]: Got next segment. Inline: true Aug 16 15:22:56.049948064 (189h56m50.481028196s) [upload=197629]: Waiting for error group managing segments... Aug 16 15:22:56.049988942 (189h56m50.481069063s) [upload=197629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:22:56.049996076 (189h56m50.481076207s) [upload=197629]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:22:56.049998711 (189h56m50.481078842s) [upload=197629]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:23:06.836570219 (189h57m1.26765035s) [upload=197629]: Done uploading segments... Aug 16 15:23:06.836588485 (189h57m1.267668606s) [upload=197629]: Upload finished. err: <nil> Duration:* 21.401609915s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 8.03796785s FAIL (2025-08-16 15:23:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 8.038001734s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XXXXXX.....................................................] duration: 5.931141994s FAIL (2025-08-16 15:23:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.93116619s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 788.572089ms FAIL (2025-08-16 15:23:11 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 788.658296ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 9.211619012s FAIL (2025-08-16 15:23:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.211646014s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [X.XXX.......................................................] duration: 732.816206ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:22:49 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 732.841766ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.X.XXX...X.......X...........X......X...X.X......X..X....] duration: 25.58876062s FAIL (2025-08-16 15:22:48 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 16 15:19:01.935812548 (189h52m56.366892679s) [upload=197555]: Starting upload Aug 16 15:19:01.935878694 (189h52m56.366958825s) [upload=197555]: Done starting upload Aug 16 15:19:01.936004836 (189h52m56.367084977s) [upload=197555]: Uploading segments... Aug 16 15:19:01.936169772 (189h52m56.367249913s) [upload=197555]: Got next segment. Inline: false Aug 16 15:19:01.936208656 (189h52m56.367288787s) [upload=197555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:19:01.936215168 (189h52m56.367295299s) [upload=197555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:19:02.436838893 (189h52m56.867919024s) [seg_pos={0 0}, upload=197555]: Begin upload segment... Aug 16 15:19:02.436886765 (189h52m56.867966896s) [seg_pos={0 0}, upload=197555]: Done begin upload segment. Aug 16 15:19:02.436907163 (189h52m56.867987294s) [noise=true , address=156.146.43.225:4466 , piece_id=2OLSHINQEE4OFYXXGBBBJZ2KG56ZEF7DZBOYYDIDEP6KUA4LT4FA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.437289305 (189h52m56.868369436s) [noise=true , address=77.90.31.12:4610 , piece_id=ALZSIIYVTIWTJYHYLG54C6L74UJJS7J7E7KGWM7NJYIIHERGPSXA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.437558330 (189h52m56.868638461s) [noise=true , address=79.127.201.215:4358 , piece_id=NEI7SIHCJZCGWGRUII5D5BK6SZQYMAEUCELGPSSYTG3EIZTCIDEA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.437911165 (189h52m56.868991296s) [noise=true , address=104.36.82.60:4343 , piece_id=F5SNMIJCO6RSAHINP7E42YHIVPMADZRNYVLJ5QFKWLCOXBSBB2JQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.438034170 (189h52m56.869114311s) [noise=true , address=66.23.193.175:4558 , piece_id=RUSF3T2RC4BMOZY3Y6ABO53FTPERYR4HMBKZFLK7PO6IC74VULQQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.438305329 (189h52m56.86938546s) [noise=true , address=104.36.82.36:4647 , piece_id=UVDSHYQLMPNLQXDIPBY4UE54ECM32C23XB7Z3ZLCUZ7X7LNMMPKQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.438742846 (189h52m56.869822977s) [noise=true , address=104.36.82.44:4891 , piece_id=WUQIJ65KSB7JXUPPT76Q4RRBLS2M2URLQOHPUYOKMI5H7EHRP4CQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.439101793 (189h52m56.870181924s) [noise=true , address=104.36.82.23:4455 , piece_id=MUTC2CTG2JJXADBW5UFDWP3HPITXDLHDUV5O66Z33IEMED5LZX4A, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.439457373 (189h52m56.870537504s) [noise=true , address=77.90.31.21:4887 , piece_id=TDBVOK2C2KXN7GXIFTWRRHWFEVGZ4BL4D2UWRDDCSIN26V2DW7PA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.439874452 (189h52m56.870954583s) [noise=true , address=104.36.82.44:4892 , piece_id=HGAGP2XS6QC36CCDLY2BZYQ72ERTIZCY2DM3IUO244JKYPFNQEPQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.440189414 (189h52m56.871269545s) [noise=true , address=104.36.82.34:4554 , piece_id=5SHIHHITT2JZYE76IK2XJSWFXZ6MPCT64VUXOZRW5YAKXOOFPIZQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.440524916 (189h52m56.871605047s) [noise=true , address=45.91.116.14:4210 , piece_id=KDO7IN4W5XK4WYBUREGFLLPYKRFMTIVUM66XAXVMGLA2AEU6DBSA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.440668481 (189h52m56.871748612s) [noise=true , address=104.36.82.17:4360 , piece_id=RZG2LMV3LKWBPO4HROH4A5ZWIMA42HJ4BIDXCIQVS6V6HXEQC2LA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.440951482 (189h52m56.872031613s) [noise=true , address=45.91.116.11:4131 , piece_id=7AX2QA3CYUHIGB2CCSTO6ZG4CLTT75KXG443LZV2OLBH2AYMUI4A, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.441195680 (189h52m56.872275811s) [noise=true , address=104.36.82.5:4843 , piece_id=L4RFYJ6FNFFE4W2SXMZM5IPGC3A33BLNZL6YOGJP4H5J3PP23IJQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.441459664 (189h52m56.872539795s) [noise=true , address=77.90.31.10:4538 , piece_id=HBIQKO5NIYWLFME3HWAFV5ZHJ66UJ6WQCK3JVKF63UD7NT3W44UQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.441719812 (189h52m56.872799943s) [noise=true , address=104.36.82.59:4310 , piece_id=VFFOACQFSN4OQHCOQ6RF4J3IY2DWCSEWVK5HDCBTVMG4HLDZH7FA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.442070373 (189h52m56.873150504s) [noise=true , address=77.90.31.11:4560 , piece_id=6BRPAUQV6XJ6HG7XH33EHJGXE3VNP7ZARJVRZ2B5ISSNLVJJIWRQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:03.644194997 (189h52m58.075275128s) [upload=197555]: Next returned nil segment Aug 16 15:19:03.644210507 (189h52m58.075290638s) [upload=197555]: Waiting for error group managing segments... Aug 16 15:19:03.967420310 (189h52m58.398500431s) [noise=true , address=66.23.193.175:4558 , piece_id=RUSF3T2RC4BMOZY3Y6ABO53FTPERYR4HMBKZFLK7PO6IC74VULQQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:04.393219317 (189h52m58.824299448s) [noise=true , address=156.146.43.225:4466 , piece_id=2OLSHINQEE4OFYXXGBBBJZ2KG56ZEF7DZBOYYDIDEP6KUA4LT4FA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:04.420543528 (189h52m58.851623659s) [noise=true , address=79.127.201.215:4358 , piece_id=NEI7SIHCJZCGWGRUII5D5BK6SZQYMAEUCELGPSSYTG3EIZTCIDEA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:04.496410662 (189h52m58.927490793s) [noise=true , address=104.36.82.34:4554 , piece_id=5SHIHHITT2JZYE76IK2XJSWFXZ6MPCT64VUXOZRW5YAKXOOFPIZQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.013496194 (189h52m59.444576325s) [noise=true , address=104.36.82.17:4360 , piece_id=RZG2LMV3LKWBPO4HROH4A5ZWIMA42HJ4BIDXCIQVS6V6HXEQC2LA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.585176407 (189h53m0.016256528s) [noise=true , address=104.36.82.60:4343 , piece_id=F5SNMIJCO6RSAHINP7E42YHIVPMADZRNYVLJ5QFKWLCOXBSBB2JQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.696057916 (189h53m0.127138047s) [noise=true , address=45.91.116.14:4210 , piece_id=KDO7IN4W5XK4WYBUREGFLLPYKRFMTIVUM66XAXVMGLA2AEU6DBSA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.771410964 (189h53m0.202491096s) [noise=true , address=77.90.31.10:4538 , piece_id=HBIQKO5NIYWLFME3HWAFV5ZHJ66UJ6WQCK3JVKF63UD7NT3W44UQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.890871970 (189h53m0.321952101s) [noise=true , address=104.36.82.5:4843 , piece_id=L4RFYJ6FNFFE4W2SXMZM5IPGC3A33BLNZL6YOGJP4H5J3PP23IJQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:06.043290113 (189h53m0.474370244s) [noise=true , address=77.90.31.12:4610 , piece_id=ALZSIIYVTIWTJYHYLG54C6L74UJJS7J7E7KGWM7NJYIIHERGPSXA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:07.255416925 (189h53m1.686497056s) [noise=true , address=104.36.82.23:4455 , piece_id=MUTC2CTG2JJXADBW5UFDWP3HPITXDLHDUV5O66Z33IEMED5LZX4A, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:07.703672319 (189h53m2.13475245s) [noise=true , address=77.90.31.11:4560 , piece_id=6BRPAUQV6XJ6HG7XH33EHJGXE3VNP7ZARJVRZ2B5ISSNLVJJIWRQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:09.663182983 (189h53m4.094263154s) [noise=true , address=45.91.116.11:4131 , piece_id=7AX2QA3CYUHIGB2CCSTO6ZG4CLTT75KXG443LZV2OLBH2AYMUI4A, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:10.138250940 (189h53m4.569331071s) [noise=true , address=104.36.82.36:4647 , piece_id=UVDSHYQLMPNLQXDIPBY4UE54ECM32C23XB7Z3ZLCUZ7X7LNMMPKQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:10.138277671 (189h53m4.569357792s) [seg_pos={0 0}, upload=197555]: Segment reached optimal threshold of 14 pieces. Aug 16 15:19:10.138630746 (189h53m4.569710877s) [noise=true , address=77.90.31.21:4887 , piece_id=TDBVOK2C2KXN7GXIFTWRRHWFEVGZ4BL4D2UWRDDCSIN26V2DW7PA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ): protocol: write tcp 188.166.225.170:47620->77.90.31.21:4887: use of closed network connection; write tcp 188.166.225.170:47620->77.90.31.21:4887: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47620->77.90.31.21:4887: use of closed network connection Aug 16 15:19:10.138675432 (189h53m4.569755563s) [noise=true , address=104.36.82.59:4310 , piece_id=VFFOACQFSN4OQHCOQ6RF4J3IY2DWCSEWVK5HDCBTVMG4HLDZH7FA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX): protocol: rpc: dial tcp 104.36.82.59:4310: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:19:10.138686132 (189h53m4.569766263s) [noise=true , address=104.36.82.44:4892 , piece_id=HGAGP2XS6QC36CCDLY2BZYQ72ERTIZCY2DM3IUO244JKYPFNQEPQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5): protocol: rpc: dial tcp 104.36.82.44:4892: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:19:10.138706822 (189h53m4.569786953s) [noise=true , address=104.36.82.44:4891 , piece_id=WUQIJ65KSB7JXUPPT76Q4RRBLS2M2URLQOHPUYOKMI5H7EHRP4CQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: rpc: dial tcp 104.36.82.44:4891: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:19:10.138723964 (189h53m4.569804095s) [seg_pos={0 0}, upload=197555]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:19:10.138777527 (189h53m4.569857658s) [upload=197555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:19:10.138781004 (189h53m4.569861135s) [upload=197555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:19:15.602831811 (189h53m10.033911942s) [upload=197555]: Done uploading segments... Aug 16 15:19:15.602860435 (189h53m10.033940556s) [upload=197555]: Upload finished. err: <nil> Duration: 25.588789206s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [X........X.....X............X...X...X..................X.X..] duration: 34.837852924s FAIL (2025-08-16 15:22:47 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 15:21:10.372800799 (189h55m4.80388092s) [upload=197590]: Starting upload Aug 16 15:21:10.372848090 (189h55m4.803928221s) [upload=197590]: Done starting upload Aug 16 15:21:10.372899549 (189h55m4.80397968s) [upload=197590]: Uploading segments... Aug 16 15:21:10.372937551 (189h55m4.804017742s) [upload=197590]: Got next segment. Inline: true Aug 16 15:21:10.372944083 (189h55m4.804024214s) [upload=197590]: Waiting for error group managing segments... Aug 16 15:21:10.372997986 (189h55m4.804078128s) [upload=197590]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:10.373004940 (189h55m4.804085131s) [upload=197590]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:10.373010090 (189h55m4.804090271s) [upload=197590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.105912913 (189h55m26.536993044s) [upload=197590]: Done uploading segments... Aug 16 15:21:32.105924454 (189h55m26.537004585s) [upload=197590]: Upload finished. err: <nil> Duration:* 34.837883142s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 2.346588681s FAIL (2025-08-16 15:22:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.346613919s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 12.990512365s FAIL (2025-08-16 15:22:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 12.990538375s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 509.870329ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:22:33 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 509.891071ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration: 8.839908148s FAIL (2025-08-16 15:22:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.839924479s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XXXXXX....................................................] duration: 6.206475198s FAIL (2025-08-16 15:22:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.206724845s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 769.180706ms FAIL (2025-08-16 15:22:26 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 769.205364ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.XXX........................................................] duration: 2.414087342s FAIL (2025-08-16 15:22:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.414115928s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 511.622182ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:22:01 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 511.644073ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XXXXX......................................................] duration: 11.119754473s FAIL (2025-08-16 15:22:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 11.119781766s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.X.XXX......................................................] duration: 855.752494ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:21:55 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 855.785236ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.X.XXX...X.......X...........X......X...X.X......X..X...] duration: 30.912432475s FAIL (2025-08-16 15:21:54 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 16 15:19:01.935812548 (189h52m56.366892679s) [upload=197555]: Starting upload Aug 16 15:19:01.935878694 (189h52m56.366958825s) [upload=197555]: Done starting upload Aug 16 15:19:01.936004836 (189h52m56.367084977s) [upload=197555]: Uploading segments... Aug 16 15:19:01.936169772 (189h52m56.367249913s) [upload=197555]: Got next segment. Inline: false Aug 16 15:19:01.936208656 (189h52m56.367288787s) [upload=197555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:19:01.936215168 (189h52m56.367295299s) [upload=197555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:19:02.436838893 (189h52m56.867919024s) [seg_pos={0 0}, upload=197555]: Begin upload segment... Aug 16 15:19:02.436886765 (189h52m56.867966896s) [seg_pos={0 0}, upload=197555]: Done begin upload segment. Aug 16 15:19:02.436907163 (189h52m56.867987294s) [noise=true , address=156.146.43.225:4466 , piece_id=2OLSHINQEE4OFYXXGBBBJZ2KG56ZEF7DZBOYYDIDEP6KUA4LT4FA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.437289305 (189h52m56.868369436s) [noise=true , address=77.90.31.12:4610 , piece_id=ALZSIIYVTIWTJYHYLG54C6L74UJJS7J7E7KGWM7NJYIIHERGPSXA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.437558330 (189h52m56.868638461s) [noise=true , address=79.127.201.215:4358 , piece_id=NEI7SIHCJZCGWGRUII5D5BK6SZQYMAEUCELGPSSYTG3EIZTCIDEA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.437911165 (189h52m56.868991296s) [noise=true , address=104.36.82.60:4343 , piece_id=F5SNMIJCO6RSAHINP7E42YHIVPMADZRNYVLJ5QFKWLCOXBSBB2JQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.438034170 (189h52m56.869114311s) [noise=true , address=66.23.193.175:4558 , piece_id=RUSF3T2RC4BMOZY3Y6ABO53FTPERYR4HMBKZFLK7PO6IC74VULQQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.438305329 (189h52m56.86938546s) [noise=true , address=104.36.82.36:4647 , piece_id=UVDSHYQLMPNLQXDIPBY4UE54ECM32C23XB7Z3ZLCUZ7X7LNMMPKQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.438742846 (189h52m56.869822977s) [noise=true , address=104.36.82.44:4891 , piece_id=WUQIJ65KSB7JXUPPT76Q4RRBLS2M2URLQOHPUYOKMI5H7EHRP4CQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.439101793 (189h52m56.870181924s) [noise=true , address=104.36.82.23:4455 , piece_id=MUTC2CTG2JJXADBW5UFDWP3HPITXDLHDUV5O66Z33IEMED5LZX4A, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.439457373 (189h52m56.870537504s) [noise=true , address=77.90.31.21:4887 , piece_id=TDBVOK2C2KXN7GXIFTWRRHWFEVGZ4BL4D2UWRDDCSIN26V2DW7PA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.439874452 (189h52m56.870954583s) [noise=true , address=104.36.82.44:4892 , piece_id=HGAGP2XS6QC36CCDLY2BZYQ72ERTIZCY2DM3IUO244JKYPFNQEPQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.440189414 (189h52m56.871269545s) [noise=true , address=104.36.82.34:4554 , piece_id=5SHIHHITT2JZYE76IK2XJSWFXZ6MPCT64VUXOZRW5YAKXOOFPIZQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.440524916 (189h52m56.871605047s) [noise=true , address=45.91.116.14:4210 , piece_id=KDO7IN4W5XK4WYBUREGFLLPYKRFMTIVUM66XAXVMGLA2AEU6DBSA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.440668481 (189h52m56.871748612s) [noise=true , address=104.36.82.17:4360 , piece_id=RZG2LMV3LKWBPO4HROH4A5ZWIMA42HJ4BIDXCIQVS6V6HXEQC2LA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.440951482 (189h52m56.872031613s) [noise=true , address=45.91.116.11:4131 , piece_id=7AX2QA3CYUHIGB2CCSTO6ZG4CLTT75KXG443LZV2OLBH2AYMUI4A, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.441195680 (189h52m56.872275811s) [noise=true , address=104.36.82.5:4843 , piece_id=L4RFYJ6FNFFE4W2SXMZM5IPGC3A33BLNZL6YOGJP4H5J3PP23IJQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.441459664 (189h52m56.872539795s) [noise=true , address=77.90.31.10:4538 , piece_id=HBIQKO5NIYWLFME3HWAFV5ZHJ66UJ6WQCK3JVKF63UD7NT3W44UQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.441719812 (189h52m56.872799943s) [noise=true , address=104.36.82.59:4310 , piece_id=VFFOACQFSN4OQHCOQ6RF4J3IY2DWCSEWVK5HDCBTVMG4HLDZH7FA, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:02.442070373 (189h52m56.873150504s) [noise=true , address=77.90.31.11:4560 , piece_id=6BRPAUQV6XJ6HG7XH33EHJGXE3VNP7ZARJVRZ2B5ISSNLVJJIWRQ, seg_pos={0 0}, upload=197555]: Uploading piece... Aug 16 15:19:03.644194997 (189h52m58.075275128s) [upload=197555]: Next returned nil segment Aug 16 15:19:03.644210507 (189h52m58.075290638s) [upload=197555]: Waiting for error group managing segments... Aug 16 15:19:03.967420310 (189h52m58.398500431s) [noise=true , address=66.23.193.175:4558 , piece_id=RUSF3T2RC4BMOZY3Y6ABO53FTPERYR4HMBKZFLK7PO6IC74VULQQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:04.393219317 (189h52m58.824299448s) [noise=true , address=156.146.43.225:4466 , piece_id=2OLSHINQEE4OFYXXGBBBJZ2KG56ZEF7DZBOYYDIDEP6KUA4LT4FA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:04.420543528 (189h52m58.851623659s) [noise=true , address=79.127.201.215:4358 , piece_id=NEI7SIHCJZCGWGRUII5D5BK6SZQYMAEUCELGPSSYTG3EIZTCIDEA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:04.496410662 (189h52m58.927490793s) [noise=true , address=104.36.82.34:4554 , piece_id=5SHIHHITT2JZYE76IK2XJSWFXZ6MPCT64VUXOZRW5YAKXOOFPIZQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.013496194 (189h52m59.444576325s) [noise=true , address=104.36.82.17:4360 , piece_id=RZG2LMV3LKWBPO4HROH4A5ZWIMA42HJ4BIDXCIQVS6V6HXEQC2LA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.585176407 (189h53m0.016256528s) [noise=true , address=104.36.82.60:4343 , piece_id=F5SNMIJCO6RSAHINP7E42YHIVPMADZRNYVLJ5QFKWLCOXBSBB2JQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.696057916 (189h53m0.127138047s) [noise=true , address=45.91.116.14:4210 , piece_id=KDO7IN4W5XK4WYBUREGFLLPYKRFMTIVUM66XAXVMGLA2AEU6DBSA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.771410964 (189h53m0.202491096s) [noise=true , address=77.90.31.10:4538 , piece_id=HBIQKO5NIYWLFME3HWAFV5ZHJ66UJ6WQCK3JVKF63UD7NT3W44UQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:05.890871970 (189h53m0.321952101s) [noise=true , address=104.36.82.5:4843 , piece_id=L4RFYJ6FNFFE4W2SXMZM5IPGC3A33BLNZL6YOGJP4H5J3PP23IJQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:06.043290113 (189h53m0.474370244s) [noise=true , address=77.90.31.12:4610 , piece_id=ALZSIIYVTIWTJYHYLG54C6L74UJJS7J7E7KGWM7NJYIIHERGPSXA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:07.255416925 (189h53m1.686497056s) [noise=true , address=104.36.82.23:4455 , piece_id=MUTC2CTG2JJXADBW5UFDWP3HPITXDLHDUV5O66Z33IEMED5LZX4A, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:07.703672319 (189h53m2.13475245s) [noise=true , address=77.90.31.11:4560 , piece_id=6BRPAUQV6XJ6HG7XH33EHJGXE3VNP7ZARJVRZ2B5ISSNLVJJIWRQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:09.663182983 (189h53m4.094263154s) [noise=true , address=45.91.116.11:4131 , piece_id=7AX2QA3CYUHIGB2CCSTO6ZG4CLTT75KXG443LZV2OLBH2AYMUI4A, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:10.138250940 (189h53m4.569331071s) [noise=true , address=104.36.82.36:4647 , piece_id=UVDSHYQLMPNLQXDIPBY4UE54ECM32C23XB7Z3ZLCUZ7X7LNMMPKQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: <nil> Aug 16 15:19:10.138277671 (189h53m4.569357792s) [seg_pos={0 0}, upload=197555]: Segment reached optimal threshold of 14 pieces. Aug 16 15:19:10.138630746 (189h53m4.569710877s) [noise=true , address=77.90.31.21:4887 , piece_id=TDBVOK2C2KXN7GXIFTWRRHWFEVGZ4BL4D2UWRDDCSIN26V2DW7PA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ): protocol: write tcp 188.166.225.170:47620->77.90.31.21:4887: use of closed network connection; write tcp 188.166.225.170:47620->77.90.31.21:4887: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47620->77.90.31.21:4887: use of closed network connection Aug 16 15:19:10.138675432 (189h53m4.569755563s) [noise=true , address=104.36.82.59:4310 , piece_id=VFFOACQFSN4OQHCOQ6RF4J3IY2DWCSEWVK5HDCBTVMG4HLDZH7FA, seg_pos={0 0}, upload=197555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX): protocol: rpc: dial tcp 104.36.82.59:4310: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:19:10.138686132 (189h53m4.569766263s) [noise=true , address=104.36.82.44:4892 , piece_id=HGAGP2XS6QC36CCDLY2BZYQ72ERTIZCY2DM3IUO244JKYPFNQEPQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5): protocol: rpc: dial tcp 104.36.82.44:4892: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:19:10.138706822 (189h53m4.569786953s) [noise=true , address=104.36.82.44:4891 , piece_id=WUQIJ65KSB7JXUPPT76Q4RRBLS2M2URLQOHPUYOKMI5H7EHRP4CQ, seg_pos={0 0}, upload=197555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: rpc: dial tcp 104.36.82.44:4891: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:19:10.138723964 (189h53m4.569804095s) [seg_pos={0 0}, upload=197555]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:19:10.138777527 (189h53m4.569857658s) [upload=197555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:19:10.138781004 (189h53m4.569861135s) [upload=197555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:19:15.602831811 (189h53m10.033911942s) [upload=197555]: Done uploading segments... Aug 16 15:19:15.602860435 (189h53m10.033940556s) [upload=197555]: Upload finished. err: <nil> Duration: 30.912455511s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 10.719615316s FAIL (2025-08-16 15:21:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 10.719641305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [.X........X.....X............X...X...X..................X.X.] duration: 11.293782121s FAIL (2025-08-16 15:21:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 15:21:10.372800799 (189h55m4.80388092s) [upload=197590]: Starting upload Aug 16 15:21:10.372848090 (189h55m4.803928221s) [upload=197590]: Done starting upload Aug 16 15:21:10.372899549 (189h55m4.80397968s) [upload=197590]: Uploading segments... Aug 16 15:21:10.372937551 (189h55m4.804017742s) [upload=197590]: Got next segment. Inline: true Aug 16 15:21:10.372944083 (189h55m4.804024214s) [upload=197590]: Waiting for error group managing segments... Aug 16 15:21:10.372997986 (189h55m4.804078128s) [upload=197590]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:10.373004940 (189h55m4.804085131s) [upload=197590]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:10.373010090 (189h55m4.804090271s) [upload=197590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.105912913 (189h55m26.536993044s) [upload=197590]: Done uploading segments... Aug 16 15:21:32.105924454 (189h55m26.537004585s) [upload=197590]: Upload finished. err: <nil> Duration:* 11.293805455s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..XXX.......................................................] duration: 2.579402068s FAIL (2025-08-16 15:21:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.579426415s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XXXXX.....................................................] duration: 18.455886905s FAIL (2025-08-16 15:21:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 18.455904237s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [..X........X.....X............X...X...X..................X.X] duration: 23.146039098s FAIL (2025-08-16 15:21:32 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 15:21:10.372800799 (189h55m4.80388092s) [upload=197590]: Starting upload Aug 16 15:21:10.372848090 (189h55m4.803928221s) [upload=197590]: Done starting upload Aug 16 15:21:10.372899549 (189h55m4.80397968s) [upload=197590]: Uploading segments... Aug 16 15:21:10.372937551 (189h55m4.804017742s) [upload=197590]: Got next segment. Inline: true Aug 16 15:21:10.372944083 (189h55m4.804024214s) [upload=197590]: Waiting for error group managing segments... Aug 16 15:21:10.372997986 (189h55m4.804078128s) [upload=197590]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:21:10.373004940 (189h55m4.804085131s) [upload=197590]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:21:10.373010090 (189h55m4.804090271s) [upload=197590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:21:32.105912913 (189h55m26.536993044s) [upload=197590]: Done uploading segments... Aug 16 15:21:32.105924454 (189h55m26.537004585s) [upload=197590]: Upload finished. err: <nil> Duration:* 23.146066311s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 501.141799ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:21:24 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 501.172136ms edge linksharing us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 766.42256ms FAIL (2025-08-16 15:21:23 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 766.451334ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.XXX........................................................] duration: 7.760923531s FAIL (2025-08-16 15:21:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 7.761010405s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..XXXXXX...................................................] duration: 6.349950505s FAIL (2025-08-16 15:21:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.349973177s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)