Orbiter History

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 327.919432ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3918ada48f420b62 FAIL (2025-11-10 01:44:15 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33468->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 327.951614ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...X..................] duration: 5.820703913s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:44:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.820735542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X....X.....................X...X..X......XX........] duration: 5.889299008s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:43:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.889328624s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...X.................] duration: 5.987938617s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:43:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.98798742s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 334.054855ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3918ada48f420b62 FAIL (2025-11-10 01:43:18 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33468->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 334.081216ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 445.09594ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3918ada48f420b62 FAIL (2025-11-10 01:42:57 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33468->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 445.123682ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...X................] duration: 5.686571798s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:42:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.686601255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X....X.....................X...X..X......XX.......] duration: 5.870496498s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:42:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.870527267s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 330.342678ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3918ada48f420b62 FAIL (2025-11-10 01:42:41 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33468->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 330.372406ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...X...............] duration: 5.825509209s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:42:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.825592537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X....X.....................X...X..X......XX......] duration: 5.792204952s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:41:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.792232815s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...X..............] duration: 5.636075959s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:41:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.636105186s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 328.641429ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3918ada48f420b62 FAIL (2025-11-10 01:41:44 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33468->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 328.670393ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X....X.....................X...X..X......XX.....] duration: 5.572663222s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:41:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.572691936s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 361.072625ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3918ada48f420b62 FAIL (2025-11-10 01:41:15 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33468->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 361.100658ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............X....X.....................X...X..X......XX....] duration: 5.954509688s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:40:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.954538172s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...X.............] duration: 5.753170957s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:40:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.753204821s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 229.573061ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3918ada48f420b62 FAIL (2025-11-10 01:40:01 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33468->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 229.6246ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...X............] duration: 5.729950828s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:39:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.729981636s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............X....X.....................X...X..X......XX...] duration: 5.844684159s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:39:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.844716251s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............X....X.....................X...X..X......XX..] duration: 5.856061739s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:39:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.856089333s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X...X...........] duration: 5.765728797s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:38:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.765760047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [................X....X.....................X...X..X......XX.] duration: 5.627183121s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:38:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.62719864s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................X....X.....................X...X..X......XX] duration: 30.246946425s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:38:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.246972925s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X...X..........] duration: 27.738167726s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:38:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 27.738197193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................X....X.....................X...X..X......X] duration: 30.256949595s trace: https://jaeger.dev.storj.io/trace/79c67906fe9a073d FAIL (2025-11-10 01:37:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 33m13.252166132s) [noise=true , address=145.79.227.13:4803 , piece_id=IGY37MK7WNG7X4YTKCVSNEGS46RDX7T4VIUNYE26YED2BUXDRHAQ, seg_pos={0 0}, upload=15480]: Uploading piece... Nov 10 01:37:14.706010593 (17h33m13.257481064s) [noise=true , address=145.79.227.8:4331 , piece_id=5Y46YUGRCAYI5WLYMT5E3UCPC4ODBVPZNAEIVELVSL2Y2DCJGF7Q, seg_pos={0 0}, upload=15480]: Uploading piece... Nov 10 01:37:14.706726445 (17h33m13.258196907s) [noise=true , address=145.79.227.13:4804 , piece_id=6FVMCECOTNEJUOYOGOYAZBG4F3WRQDO2LFR2E6KYLDGP7F2RFTUQ, seg_pos={0 0}, upload=15480]: Uploading piece... Nov 10 01:37:14.707257767 (17h33m13.258728228s) [noise=true , address=77.90.31.11:4555 , piece_id=YJPK2S26LYLC4N7ZDMUEDPMJPGYXLPEU2E6C35AHGSPLIULY2ZEQ, seg_pos={0 0}, upload=15480]: Uploading piece... Nov 10 01:37:14.707774912 (17h33m13.259245373s) [noise=true , address=45.91.116.4:4940 , piece_id=AUL7NTE37HVF6OWM2HAV3KAPFRCZCKEUAJORBDYZLIFMOF5GEQ3A, seg_pos={0 0}, upload=15480]: Uploading piece... Nov 10 01:37:14.708403529 (17h33m13.25987399s) [noise=true , address=145.79.227.28:4727 , piece_id=LPQLM4HNIXV2OYNCLFPC3SSADG7VCYVUNXWW5WMBFQXHPZG26WBA, seg_pos={0 0}, upload=15480]: Uploading piece... Nov 10 01:37:16.472679167 (17h33m15.024149618s) [upload=15480]: Next returned nil segment Nov 10 01:37:16.472715516 (17h33m15.024185967s) [upload=15480]: Waiting for error group managing segments... Nov 10 01:37:16.816175472 (17h33m15.367645933s) [noise=true , address=77.90.31.11:4555 , piece_id=YJPK2S26LYLC4N7ZDMUEDPMJPGYXLPEU2E6C35AHGSPLIULY2ZEQ, seg_pos={0 0}, upload=15480]: Done uploading piece. err: <nil> Nov 10 01:37:16.839569008 (17h33m15.391039469s) [noise=true , address=77.90.31.13:4627 , piece_id=F3AZJB2AZMSZTVHUZWXDKCTKARVE5VBFADEVPXRRN2DITBK577AA, seg_pos={0 0}, upload=15480]: Done uploading piece. err: <nil> Nov 10 01:37:16.903847978 (17h33m15.455318439s) [noise=true , address=45.91.116.4:4940 , piece_id=AUL7NTE37HVF6OWM2HAV3KAPFRCZCKEUAJORBDYZLIFMOF5GEQ3A, seg_pos={0 0}, upload=15480]: Done uploading piece. err: <nil> Nov 10 01:37:37.097333578 (17h33m35.648804039s) [noise=true , address=145.79.227.10:4401 , piece_id=QUXBSRXGOOFQJHMNBYPLR62ZW4ZJDNMX7GS62EEEGEZKCNBNVEWA, seg_pos={0 0}, upload=15480]: Done uploading piece. err: <nil> Nov 10 01:37:43.738443057 (17h33m42.289913508s) [noise=true , address=145.79.227.13:4803 , piece_id=IGY37MK7WNG7X4YTKCVSNEGS46RDX7T4VIUNYE26YED2BUXDRHAQ, seg_pos={0 0}, upload=15480]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:58048->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:58048->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58048->145.79.227.13:4803: use of closed network connection Nov 10 01:37:43.738490156 (17h33m42.289960618s) [noise=true , address=145.79.227.14:4838 , piece_id=2QQUF5ZKNXA4YNEVVTKJEBQV4T3IGM5B3RBRUI5UT3YZWQBK4SXQ, seg_pos={0 0}, upload=15480]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: write tcp 188.166.225.170:53090->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:53090->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53090->145.79.227.14:4838: use of closed network connection Nov 10 01:37:43.738661763 (17h33m42.290132224s) [noise=true , address=145.79.227.9:4365 , piece_id=YF5WJHNJ3CSK5FBTSXYYDB2L47IDVDQXINTK4YWKQTPYVN35XP7A, seg_pos={0 0}, upload=15480]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:37544->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:37544->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37544->145.79.227.9:4365: use of closed network connection Nov 10 01:37:43.738758568 (17h33m42.290229029s) [noise=true , address=145.79.227.28:4727 , piece_id=LPQLM4HNIXV2OYNCLFPC3SSADG7VCYVUNXWW5WMBFQXHPZG26WBA, seg_pos={0 0}, upload=15480]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:55620->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:55620->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55620->145.79.227.28:4727: use of closed network connection Nov 10 01:37:43.738852045 (17h33m42.290322506s) [noise=true , address=145.79.227.11:4731 , piece_id=YXGUJ5ITI4DVC2AZNDIVTL7WF34AOSOSENA366SCDIDD7TP3KECA, seg_pos={0 0}, upload=15480]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:57666->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:57666->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57666->145.79.227.11:4731: use of closed network connection Nov 10 01:37:43.739067536 (17h33m42.290537997s) [noise=true , address=145.79.227.26:4655 , piece_id=CHQWMFM2ISLLGZA7P3ZXQKFAJDOKIOR2VDE2ZARKBSABUWFKHWNQ, seg_pos={0 0}, upload=15480]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:37868->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:37868->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37868->145.79.227.26:4655: use of closed network connection Nov 10 01:37:43.739128111 (17h33m42.290598562s) [noise=true , address=145.79.227.8:4331 , piece_id=5Y46YUGRCAYI5WLYMT5E3UCPC4ODBVPZNAEIVELVSL2Y2DCJGF7Q, seg_pos={0 0}, upload=15480]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:39332->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:39332->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39332->145.79.227.8:4331: use of closed network connection Nov 10 01:37:43.739176764 (17h33m42.290647225s) [noise=true , address=145.79.227.28:4726 , piece_id=VYDWCJHDJXX2D77GXYC24FDNCPUKG3ZCLY7XBJUOF7F24HSYWWKQ, seg_pos={0 0}, upload=15480]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:45590->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:45590->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45590->145.79.227.28:4726: use of closed network connection Nov 10 01:37:43.739289659 (17h33m42.29076012s) [noise=true , address=145.79.227.16:4910 , piece_id=EAPGQYBDAWOOZHIZCQ76TEUNF6QNZTWRCCCPRCWUWEFILVWNDMJA, seg_pos={0 0}, upload=15480]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:40094->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:40094->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40094->145.79.227.16:4910: use of closed network connection Nov 10 01:37:43.739469621 (17h33m42.290940113s) [noise=true , address=145.79.227.27:4691 , piece_id=JAWH2HUTLPKHAOWI25QAT2MU6HVQXTCU2UMJLVPDF4S757FQIIAQ, seg_pos={0 0}, upload=15480]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:51234->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:51234->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51234->145.79.227.27:4691: use of closed network connection Nov 10 01:37:43.739583458 (17h33m42.291053919s) [noise=true , address=145.79.227.13:4804 , piece_id=6FVMCECOTNEJUOYOGOYAZBG4F3WRQDO2LFR2E6KYLDGP7F2RFTUQ, seg_pos={0 0}, upload=15480]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:44898->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:44898->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44898->145.79.227.13:4804: use of closed network connection Nov 10 01:37:43.739778971 (17h33m42.291249432s) [noise=true , address=145.79.227.20:4437 , piece_id=4IVXNWVGZKIPMMCGRSVKITZUPIELGWKVGZSBCCC3ANFACUI2BCIQ, seg_pos={0 0}, upload=15480]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:52992->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:37:43.739809950 (17h33m42.291280401s) [seg_pos={0 0}, upload=15480]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 10 01:37:43.993520795 (17h33m42.544991246s) [upload=15480]: Done uploading segments... Nov 10 01:37:43.993561652 (17h33m42.545032113s) [upload=15480]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.256972008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X...X.........] duration: 6.638708142s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:37:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.638745192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................X....X.....................X...X..X......] duration: 5.964076627s trace: https://jaeger.dev.storj.io/trace/51b7178e105d4206 FAIL (2025-11-10 01:36:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:32:51.801397615 (17h28m50.352868076s) [upload=15416]: Starting upload Nov 10 01:32:51.801485843 (17h28m50.352956305s) [upload=15416]: Done starting upload Nov 10 01:32:51.801680303 (17h28m50.353150764s) [upload=15416]: Uploading segments... Nov 10 01:32:51.801845388 (17h28m50.353315839s) [upload=15416]: Got next segment. Inline: false Nov 10 01:32:51.801927615 (17h28m50.353398076s) [upload=15416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:32:51.801935770 (17h28m50.353406231s) [upload=15416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:32:52.053537580 (17h28m50.605008041s) [seg_pos={0 0}, upload=15416]: Begin upload segment... Nov 10 01:32:52.053623583 (17h28m50.605094044s) [seg_pos={0 0}, upload=15416]: Done begin upload segment. Nov 10 01:32:52.053688186 (17h28m50.605158637s) [noise=true , address=145.79.227.13:4802 , piece_id=FQ6JKNKMAMV5PVGMA5XTX7AVMJNL4OCF353IJQGRU72KT3UIK7FA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.053715909 (17h28m50.60518637s) [noise=true , address=145.79.227.10:4403 , piece_id=CJLGVRMRLLVE2G7KDO3HA22ARYBAPQ63CKVXYG7OXH7B6U7DPMYA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.054433696 (17h28m50.605904158s) [noise=true , address=145.79.227.25:4618 , piece_id=WGVJKWE4FJO6QFKPXMZ4QUYY5H6FLKXLXHU7N2VCIPJI5GXWBN5Q, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.054862843 (17h28m50.606333294s) [noise=true , address=145.79.227.8:4331 , piece_id=J5F7ATSE3DRAEDL6VS4OSQLE5UJSJKTB56DDQSJANKPIHCECS7QQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.055873908 (17h28m50.60734436s) [noise=true , address=145.79.227.32:4836 , piece_id=I47WTHOTKBERKTSUXEOYFKDSL22ESSBBFGHB7Z5QGHFII3NLUXUA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.056980796 (17h28m50.608451257s) [noise=true , address=145.79.227.13:4803 , piece_id=NJUXOTFWDS5NSHJTTTW444UAYUKMEJ4OA26GPJWOXFEGUW7MZN2A, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.057477022 (17h28m50.608947483s) [noise=true , address=145.79.227.9:4366 , piece_id=4YDVA4RMRUQJSR7JJZOXOYHPRCOCS75JIVSFSI7MV3UFOUY3E7OQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.058018342 (17h28m50.609488803s) [noise=true , address=145.79.227.28:4726 , piece_id=UNWVLOG6RB4BPIFB32Q5HPQ52DSHYEZ6FOYWELSDBCZB23A6AFAA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.060006197 (17h28m50.611476658s) [noise=true , address=145.79.227.21:4474 , piece_id=2W5SD2DUKNV6OQ2NK3Z4MRYMW2WTXEJ66SOQR6QK6LUOSF4XR6KA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.060567757 (17h28m50.612038218s) [noise=true , address=145.79.227.27:4691 , piece_id=5GBTJKY553CPE7EKYNIQEKVLBKUTWDBY6UZZ2TO4FQ3T3ELDE2QA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.062232697 (17h28m50.613703148s) [noise=true , address=145.79.227.8:4330 , piece_id=WO54DON3YBUKHQDWVVZUAFL3V3QCEBGEP6URNAOOFGFGNBCX33WA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.064096336 (17h28m50.615566797s) [noise=true , address=145.79.227.14:4837 , piece_id=G4RWP56E2WEPB6U6OVTPO4JS4AL2OICLX7TVTD2G3R6SATRA6GKQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.065772528 (17h28m50.617242989s) [noise=true , address=145.79.227.6:4258 , piece_id=ZPMTRI273KK7JZRSLPDMZMYTYH6VKIRMCX4IWT6GW2XNXENRKCRA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.067776333 (17h28m50.619246794s) [noise=true , address=77.90.31.12:4590 , piece_id=Y6N4IUYUMGJDFQ5EWS4X6UM7UPDFGNAZGOSV6DSVGNMYEPTPET6Q, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.069763207 (17h28m50.621233668s) [noise=true , address=145.79.227.16:4910 , piece_id=52L4KWIFZ4JWJ3DFSJZ33GAPGWMTUF22455PAJGZOQOEFCR3DOBQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.071712599 (17h28m50.62318309s) [noise=true , address=77.90.31.11:4554 , piece_id=4WL2ZCU3QIEEP4FFBOREJWIGL5QX2OHXZ2SHW4CQHW7CPCFLWSOA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:53.794020774 (17h28m52.345491235s) [upload=15416]: Next returned nil segment Nov 10 01:32:53.794034640 (17h28m52.345505101s) [upload=15416]: Waiting for error group managing segments... Nov 10 01:32:54.124111762 (17h28m52.675582223s) [noise=true , address=77.90.31.12:4590 , piece_id=Y6N4IUYUMGJDFQ5EWS4X6UM7UPDFGNAZGOSV6DSVGNMYEPTPET6Q, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:32:54.204858650 (17h28m52.756329101s) [noise=true , address=77.90.31.11:4554 , piece_id=4WL2ZCU3QIEEP4FFBOREJWIGL5QX2OHXZ2SHW4CQHW7CPCFLWSOA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:01.893062231 (17h29m0.444532702s) [noise=true , address=145.79.227.8:4331 , piece_id=J5F7ATSE3DRAEDL6VS4OSQLE5UJSJKTB56DDQSJANKPIHCECS7QQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:08.535423736 (17h29m7.086894197s) [noise=true , address=145.79.227.6:4258 , piece_id=ZPMTRI273KK7JZRSLPDMZMYTYH6VKIRMCX4IWT6GW2XNXENRKCRA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:09.147601053 (17h29m7.699071514s) [noise=true , address=145.79.227.28:4726 , piece_id=UNWVLOG6RB4BPIFB32Q5HPQ52DSHYEZ6FOYWELSDBCZB23A6AFAA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:10.101167260 (17h29m8.652637722s) [noise=true , address=145.79.227.9:4366 , piece_id=4YDVA4RMRUQJSR7JJZOXOYHPRCOCS75JIVSFSI7MV3UFOUY3E7OQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:10.453714069 (17h29m9.00518453s) [noise=true , address=145.79.227.16:4910 , piece_id=52L4KWIFZ4JWJ3DFSJZ33GAPGWMTUF22455PAJGZOQOEFCR3DOBQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:11.100100727 (17h29m9.651571188s) [noise=true , address=145.79.227.10:4403 , piece_id=CJLGVRMRLLVE2G7KDO3HA22ARYBAPQ63CKVXYG7OXH7B6U7DPMYA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:14.822230635 (17h29m13.373701096s) [noise=true , address=145.79.227.21:4474 , piece_id=2W5SD2DUKNV6OQ2NK3Z4MRYMW2WTXEJ66SOQR6QK6LUOSF4XR6KA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:17.170665030 (17h29m15.722135481s) [noise=true , address=145.79.227.27:4691 , piece_id=5GBTJKY553CPE7EKYNIQEKVLBKUTWDBY6UZZ2TO4FQ3T3ELDE2QA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:19.422554030 (17h29m17.974024491s) [noise=true , address=145.79.227.13:4803 , piece_id=NJUXOTFWDS5NSHJTTTW444UAYUKMEJ4OA26GPJWOXFEGUW7MZN2A, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:21.096313239 (17h29m19.6477837s) [noise=true , address=145.79.227.32:4836 , piece_id=I47WTHOTKBERKTSUXEOYFKDSL22ESSBBFGHB7Z5QGHFII3NLUXUA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096488773 (17h29m19.647959234s) [noise=true , address=145.79.227.8:4330 , piece_id=WO54DON3YBUKHQDWVVZUAFL3V3QCEBGEP6URNAOOFGFGNBCX33WA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096644479 (17h29m19.64811493s) [noise=true , address=145.79.227.13:4802 , piece_id=FQ6JKNKMAMV5PVGMA5XTX7AVMJNL4OCF353IJQGRU72KT3UIK7FA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096788984 (17h29m19.648259445s) [noise=true , address=145.79.227.25:4618 , piece_id=WGVJKWE4FJO6QFKPXMZ4QUYY5H6FLKXLXHU7N2VCIPJI5GXWBN5Q, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096954390 (17h29m19.648424861s) [noise=true , address=145.79.227.14:4837 , piece_id=G4RWP56E2WEPB6U6OVTPO4JS4AL2OICLX7TVTD2G3R6SATRA6GKQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection Nov 10 01:33:21.096989987 (17h29m19.648460448s) [seg_pos={0 0}, upload=15416]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:33:21.342594608 (17h29m19.894065069s) [upload=15416]: Done uploading segments... Nov 10 01:33:21.342640315 (17h29m19.894110766s) [upload=15416]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.964107105s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...X........] duration: 5.69270384s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:36:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.692735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X....X.....................X...X..X.....] duration: 5.740061669s trace: https://jaeger.dev.storj.io/trace/51b7178e105d4206 FAIL (2025-11-10 01:36:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:32:51.801397615 (17h28m50.352868076s) [upload=15416]: Starting upload Nov 10 01:32:51.801485843 (17h28m50.352956305s) [upload=15416]: Done starting upload Nov 10 01:32:51.801680303 (17h28m50.353150764s) [upload=15416]: Uploading segments... Nov 10 01:32:51.801845388 (17h28m50.353315839s) [upload=15416]: Got next segment. Inline: false Nov 10 01:32:51.801927615 (17h28m50.353398076s) [upload=15416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:32:51.801935770 (17h28m50.353406231s) [upload=15416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:32:52.053537580 (17h28m50.605008041s) [seg_pos={0 0}, upload=15416]: Begin upload segment... Nov 10 01:32:52.053623583 (17h28m50.605094044s) [seg_pos={0 0}, upload=15416]: Done begin upload segment. Nov 10 01:32:52.053688186 (17h28m50.605158637s) [noise=true , address=145.79.227.13:4802 , piece_id=FQ6JKNKMAMV5PVGMA5XTX7AVMJNL4OCF353IJQGRU72KT3UIK7FA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.053715909 (17h28m50.60518637s) [noise=true , address=145.79.227.10:4403 , piece_id=CJLGVRMRLLVE2G7KDO3HA22ARYBAPQ63CKVXYG7OXH7B6U7DPMYA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.054433696 (17h28m50.605904158s) [noise=true , address=145.79.227.25:4618 , piece_id=WGVJKWE4FJO6QFKPXMZ4QUYY5H6FLKXLXHU7N2VCIPJI5GXWBN5Q, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.054862843 (17h28m50.606333294s) [noise=true , address=145.79.227.8:4331 , piece_id=J5F7ATSE3DRAEDL6VS4OSQLE5UJSJKTB56DDQSJANKPIHCECS7QQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.055873908 (17h28m50.60734436s) [noise=true , address=145.79.227.32:4836 , piece_id=I47WTHOTKBERKTSUXEOYFKDSL22ESSBBFGHB7Z5QGHFII3NLUXUA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.056980796 (17h28m50.608451257s) [noise=true , address=145.79.227.13:4803 , piece_id=NJUXOTFWDS5NSHJTTTW444UAYUKMEJ4OA26GPJWOXFEGUW7MZN2A, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.057477022 (17h28m50.608947483s) [noise=true , address=145.79.227.9:4366 , piece_id=4YDVA4RMRUQJSR7JJZOXOYHPRCOCS75JIVSFSI7MV3UFOUY3E7OQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.058018342 (17h28m50.609488803s) [noise=true , address=145.79.227.28:4726 , piece_id=UNWVLOG6RB4BPIFB32Q5HPQ52DSHYEZ6FOYWELSDBCZB23A6AFAA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.060006197 (17h28m50.611476658s) [noise=true , address=145.79.227.21:4474 , piece_id=2W5SD2DUKNV6OQ2NK3Z4MRYMW2WTXEJ66SOQR6QK6LUOSF4XR6KA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.060567757 (17h28m50.612038218s) [noise=true , address=145.79.227.27:4691 , piece_id=5GBTJKY553CPE7EKYNIQEKVLBKUTWDBY6UZZ2TO4FQ3T3ELDE2QA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.062232697 (17h28m50.613703148s) [noise=true , address=145.79.227.8:4330 , piece_id=WO54DON3YBUKHQDWVVZUAFL3V3QCEBGEP6URNAOOFGFGNBCX33WA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.064096336 (17h28m50.615566797s) [noise=true , address=145.79.227.14:4837 , piece_id=G4RWP56E2WEPB6U6OVTPO4JS4AL2OICLX7TVTD2G3R6SATRA6GKQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.065772528 (17h28m50.617242989s) [noise=true , address=145.79.227.6:4258 , piece_id=ZPMTRI273KK7JZRSLPDMZMYTYH6VKIRMCX4IWT6GW2XNXENRKCRA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.067776333 (17h28m50.619246794s) [noise=true , address=77.90.31.12:4590 , piece_id=Y6N4IUYUMGJDFQ5EWS4X6UM7UPDFGNAZGOSV6DSVGNMYEPTPET6Q, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.069763207 (17h28m50.621233668s) [noise=true , address=145.79.227.16:4910 , piece_id=52L4KWIFZ4JWJ3DFSJZ33GAPGWMTUF22455PAJGZOQOEFCR3DOBQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.071712599 (17h28m50.62318309s) [noise=true , address=77.90.31.11:4554 , piece_id=4WL2ZCU3QIEEP4FFBOREJWIGL5QX2OHXZ2SHW4CQHW7CPCFLWSOA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:53.794020774 (17h28m52.345491235s) [upload=15416]: Next returned nil segment Nov 10 01:32:53.794034640 (17h28m52.345505101s) [upload=15416]: Waiting for error group managing segments... Nov 10 01:32:54.124111762 (17h28m52.675582223s) [noise=true , address=77.90.31.12:4590 , piece_id=Y6N4IUYUMGJDFQ5EWS4X6UM7UPDFGNAZGOSV6DSVGNMYEPTPET6Q, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:32:54.204858650 (17h28m52.756329101s) [noise=true , address=77.90.31.11:4554 , piece_id=4WL2ZCU3QIEEP4FFBOREJWIGL5QX2OHXZ2SHW4CQHW7CPCFLWSOA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:01.893062231 (17h29m0.444532702s) [noise=true , address=145.79.227.8:4331 , piece_id=J5F7ATSE3DRAEDL6VS4OSQLE5UJSJKTB56DDQSJANKPIHCECS7QQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:08.535423736 (17h29m7.086894197s) [noise=true , address=145.79.227.6:4258 , piece_id=ZPMTRI273KK7JZRSLPDMZMYTYH6VKIRMCX4IWT6GW2XNXENRKCRA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:09.147601053 (17h29m7.699071514s) [noise=true , address=145.79.227.28:4726 , piece_id=UNWVLOG6RB4BPIFB32Q5HPQ52DSHYEZ6FOYWELSDBCZB23A6AFAA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:10.101167260 (17h29m8.652637722s) [noise=true , address=145.79.227.9:4366 , piece_id=4YDVA4RMRUQJSR7JJZOXOYHPRCOCS75JIVSFSI7MV3UFOUY3E7OQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:10.453714069 (17h29m9.00518453s) [noise=true , address=145.79.227.16:4910 , piece_id=52L4KWIFZ4JWJ3DFSJZ33GAPGWMTUF22455PAJGZOQOEFCR3DOBQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:11.100100727 (17h29m9.651571188s) [noise=true , address=145.79.227.10:4403 , piece_id=CJLGVRMRLLVE2G7KDO3HA22ARYBAPQ63CKVXYG7OXH7B6U7DPMYA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:14.822230635 (17h29m13.373701096s) [noise=true , address=145.79.227.21:4474 , piece_id=2W5SD2DUKNV6OQ2NK3Z4MRYMW2WTXEJ66SOQR6QK6LUOSF4XR6KA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:17.170665030 (17h29m15.722135481s) [noise=true , address=145.79.227.27:4691 , piece_id=5GBTJKY553CPE7EKYNIQEKVLBKUTWDBY6UZZ2TO4FQ3T3ELDE2QA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:19.422554030 (17h29m17.974024491s) [noise=true , address=145.79.227.13:4803 , piece_id=NJUXOTFWDS5NSHJTTTW444UAYUKMEJ4OA26GPJWOXFEGUW7MZN2A, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:21.096313239 (17h29m19.6477837s) [noise=true , address=145.79.227.32:4836 , piece_id=I47WTHOTKBERKTSUXEOYFKDSL22ESSBBFGHB7Z5QGHFII3NLUXUA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096488773 (17h29m19.647959234s) [noise=true , address=145.79.227.8:4330 , piece_id=WO54DON3YBUKHQDWVVZUAFL3V3QCEBGEP6URNAOOFGFGNBCX33WA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096644479 (17h29m19.64811493s) [noise=true , address=145.79.227.13:4802 , piece_id=FQ6JKNKMAMV5PVGMA5XTX7AVMJNL4OCF353IJQGRU72KT3UIK7FA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096788984 (17h29m19.648259445s) [noise=true , address=145.79.227.25:4618 , piece_id=WGVJKWE4FJO6QFKPXMZ4QUYY5H6FLKXLXHU7N2VCIPJI5GXWBN5Q, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096954390 (17h29m19.648424861s) [noise=true , address=145.79.227.14:4837 , piece_id=G4RWP56E2WEPB6U6OVTPO4JS4AL2OICLX7TVTD2G3R6SATRA6GKQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection Nov 10 01:33:21.096989987 (17h29m19.648460448s) [seg_pos={0 0}, upload=15416]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:33:21.342594608 (17h29m19.894065069s) [upload=15416]: Done uploading segments... Nov 10 01:33:21.342640315 (17h29m19.894110766s) [upload=15416]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.740091967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X...X.......] duration: 5.594976099s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:35:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.595008791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................X....X.....................X...X..X....] duration: 5.774380423s trace: https://jaeger.dev.storj.io/trace/51b7178e105d4206 FAIL (2025-11-10 01:35:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:32:51.801397615 (17h28m50.352868076s) [upload=15416]: Starting upload Nov 10 01:32:51.801485843 (17h28m50.352956305s) [upload=15416]: Done starting upload Nov 10 01:32:51.801680303 (17h28m50.353150764s) [upload=15416]: Uploading segments... Nov 10 01:32:51.801845388 (17h28m50.353315839s) [upload=15416]: Got next segment. Inline: false Nov 10 01:32:51.801927615 (17h28m50.353398076s) [upload=15416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:32:51.801935770 (17h28m50.353406231s) [upload=15416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:32:52.053537580 (17h28m50.605008041s) [seg_pos={0 0}, upload=15416]: Begin upload segment... Nov 10 01:32:52.053623583 (17h28m50.605094044s) [seg_pos={0 0}, upload=15416]: Done begin upload segment. Nov 10 01:32:52.053688186 (17h28m50.605158637s) [noise=true , address=145.79.227.13:4802 , piece_id=FQ6JKNKMAMV5PVGMA5XTX7AVMJNL4OCF353IJQGRU72KT3UIK7FA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.053715909 (17h28m50.60518637s) [noise=true , address=145.79.227.10:4403 , piece_id=CJLGVRMRLLVE2G7KDO3HA22ARYBAPQ63CKVXYG7OXH7B6U7DPMYA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.054433696 (17h28m50.605904158s) [noise=true , address=145.79.227.25:4618 , piece_id=WGVJKWE4FJO6QFKPXMZ4QUYY5H6FLKXLXHU7N2VCIPJI5GXWBN5Q, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.054862843 (17h28m50.606333294s) [noise=true , address=145.79.227.8:4331 , piece_id=J5F7ATSE3DRAEDL6VS4OSQLE5UJSJKTB56DDQSJANKPIHCECS7QQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.055873908 (17h28m50.60734436s) [noise=true , address=145.79.227.32:4836 , piece_id=I47WTHOTKBERKTSUXEOYFKDSL22ESSBBFGHB7Z5QGHFII3NLUXUA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.056980796 (17h28m50.608451257s) [noise=true , address=145.79.227.13:4803 , piece_id=NJUXOTFWDS5NSHJTTTW444UAYUKMEJ4OA26GPJWOXFEGUW7MZN2A, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.057477022 (17h28m50.608947483s) [noise=true , address=145.79.227.9:4366 , piece_id=4YDVA4RMRUQJSR7JJZOXOYHPRCOCS75JIVSFSI7MV3UFOUY3E7OQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.058018342 (17h28m50.609488803s) [noise=true , address=145.79.227.28:4726 , piece_id=UNWVLOG6RB4BPIFB32Q5HPQ52DSHYEZ6FOYWELSDBCZB23A6AFAA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.060006197 (17h28m50.611476658s) [noise=true , address=145.79.227.21:4474 , piece_id=2W5SD2DUKNV6OQ2NK3Z4MRYMW2WTXEJ66SOQR6QK6LUOSF4XR6KA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.060567757 (17h28m50.612038218s) [noise=true , address=145.79.227.27:4691 , piece_id=5GBTJKY553CPE7EKYNIQEKVLBKUTWDBY6UZZ2TO4FQ3T3ELDE2QA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.062232697 (17h28m50.613703148s) [noise=true , address=145.79.227.8:4330 , piece_id=WO54DON3YBUKHQDWVVZUAFL3V3QCEBGEP6URNAOOFGFGNBCX33WA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.064096336 (17h28m50.615566797s) [noise=true , address=145.79.227.14:4837 , piece_id=G4RWP56E2WEPB6U6OVTPO4JS4AL2OICLX7TVTD2G3R6SATRA6GKQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.065772528 (17h28m50.617242989s) [noise=true , address=145.79.227.6:4258 , piece_id=ZPMTRI273KK7JZRSLPDMZMYTYH6VKIRMCX4IWT6GW2XNXENRKCRA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.067776333 (17h28m50.619246794s) [noise=true , address=77.90.31.12:4590 , piece_id=Y6N4IUYUMGJDFQ5EWS4X6UM7UPDFGNAZGOSV6DSVGNMYEPTPET6Q, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.069763207 (17h28m50.621233668s) [noise=true , address=145.79.227.16:4910 , piece_id=52L4KWIFZ4JWJ3DFSJZ33GAPGWMTUF22455PAJGZOQOEFCR3DOBQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.071712599 (17h28m50.62318309s) [noise=true , address=77.90.31.11:4554 , piece_id=4WL2ZCU3QIEEP4FFBOREJWIGL5QX2OHXZ2SHW4CQHW7CPCFLWSOA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:53.794020774 (17h28m52.345491235s) [upload=15416]: Next returned nil segment Nov 10 01:32:53.794034640 (17h28m52.345505101s) [upload=15416]: Waiting for error group managing segments... Nov 10 01:32:54.124111762 (17h28m52.675582223s) [noise=true , address=77.90.31.12:4590 , piece_id=Y6N4IUYUMGJDFQ5EWS4X6UM7UPDFGNAZGOSV6DSVGNMYEPTPET6Q, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:32:54.204858650 (17h28m52.756329101s) [noise=true , address=77.90.31.11:4554 , piece_id=4WL2ZCU3QIEEP4FFBOREJWIGL5QX2OHXZ2SHW4CQHW7CPCFLWSOA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:01.893062231 (17h29m0.444532702s) [noise=true , address=145.79.227.8:4331 , piece_id=J5F7ATSE3DRAEDL6VS4OSQLE5UJSJKTB56DDQSJANKPIHCECS7QQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:08.535423736 (17h29m7.086894197s) [noise=true , address=145.79.227.6:4258 , piece_id=ZPMTRI273KK7JZRSLPDMZMYTYH6VKIRMCX4IWT6GW2XNXENRKCRA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:09.147601053 (17h29m7.699071514s) [noise=true , address=145.79.227.28:4726 , piece_id=UNWVLOG6RB4BPIFB32Q5HPQ52DSHYEZ6FOYWELSDBCZB23A6AFAA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:10.101167260 (17h29m8.652637722s) [noise=true , address=145.79.227.9:4366 , piece_id=4YDVA4RMRUQJSR7JJZOXOYHPRCOCS75JIVSFSI7MV3UFOUY3E7OQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:10.453714069 (17h29m9.00518453s) [noise=true , address=145.79.227.16:4910 , piece_id=52L4KWIFZ4JWJ3DFSJZ33GAPGWMTUF22455PAJGZOQOEFCR3DOBQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:11.100100727 (17h29m9.651571188s) [noise=true , address=145.79.227.10:4403 , piece_id=CJLGVRMRLLVE2G7KDO3HA22ARYBAPQ63CKVXYG7OXH7B6U7DPMYA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:14.822230635 (17h29m13.373701096s) [noise=true , address=145.79.227.21:4474 , piece_id=2W5SD2DUKNV6OQ2NK3Z4MRYMW2WTXEJ66SOQR6QK6LUOSF4XR6KA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:17.170665030 (17h29m15.722135481s) [noise=true , address=145.79.227.27:4691 , piece_id=5GBTJKY553CPE7EKYNIQEKVLBKUTWDBY6UZZ2TO4FQ3T3ELDE2QA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:19.422554030 (17h29m17.974024491s) [noise=true , address=145.79.227.13:4803 , piece_id=NJUXOTFWDS5NSHJTTTW444UAYUKMEJ4OA26GPJWOXFEGUW7MZN2A, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:21.096313239 (17h29m19.6477837s) [noise=true , address=145.79.227.32:4836 , piece_id=I47WTHOTKBERKTSUXEOYFKDSL22ESSBBFGHB7Z5QGHFII3NLUXUA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096488773 (17h29m19.647959234s) [noise=true , address=145.79.227.8:4330 , piece_id=WO54DON3YBUKHQDWVVZUAFL3V3QCEBGEP6URNAOOFGFGNBCX33WA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096644479 (17h29m19.64811493s) [noise=true , address=145.79.227.13:4802 , piece_id=FQ6JKNKMAMV5PVGMA5XTX7AVMJNL4OCF353IJQGRU72KT3UIK7FA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096788984 (17h29m19.648259445s) [noise=true , address=145.79.227.25:4618 , piece_id=WGVJKWE4FJO6QFKPXMZ4QUYY5H6FLKXLXHU7N2VCIPJI5GXWBN5Q, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096954390 (17h29m19.648424861s) [noise=true , address=145.79.227.14:4837 , piece_id=G4RWP56E2WEPB6U6OVTPO4JS4AL2OICLX7TVTD2G3R6SATRA6GKQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection Nov 10 01:33:21.096989987 (17h29m19.648460448s) [seg_pos={0 0}, upload=15416]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:33:21.342594608 (17h29m19.894065069s) [upload=15416]: Done uploading segments... Nov 10 01:33:21.342640315 (17h29m19.894110766s) [upload=15416]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.774417946s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................X....X.....................X...X..X...] duration: 5.747311262s trace: https://jaeger.dev.storj.io/trace/51b7178e105d4206 FAIL (2025-11-10 01:34:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:32:51.801397615 (17h28m50.352868076s) [upload=15416]: Starting upload Nov 10 01:32:51.801485843 (17h28m50.352956305s) [upload=15416]: Done starting upload Nov 10 01:32:51.801680303 (17h28m50.353150764s) [upload=15416]: Uploading segments... Nov 10 01:32:51.801845388 (17h28m50.353315839s) [upload=15416]: Got next segment. Inline: false Nov 10 01:32:51.801927615 (17h28m50.353398076s) [upload=15416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:32:51.801935770 (17h28m50.353406231s) [upload=15416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:32:52.053537580 (17h28m50.605008041s) [seg_pos={0 0}, upload=15416]: Begin upload segment... Nov 10 01:32:52.053623583 (17h28m50.605094044s) [seg_pos={0 0}, upload=15416]: Done begin upload segment. Nov 10 01:32:52.053688186 (17h28m50.605158637s) [noise=true , address=145.79.227.13:4802 , piece_id=FQ6JKNKMAMV5PVGMA5XTX7AVMJNL4OCF353IJQGRU72KT3UIK7FA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.053715909 (17h28m50.60518637s) [noise=true , address=145.79.227.10:4403 , piece_id=CJLGVRMRLLVE2G7KDO3HA22ARYBAPQ63CKVXYG7OXH7B6U7DPMYA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.054433696 (17h28m50.605904158s) [noise=true , address=145.79.227.25:4618 , piece_id=WGVJKWE4FJO6QFKPXMZ4QUYY5H6FLKXLXHU7N2VCIPJI5GXWBN5Q, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.054862843 (17h28m50.606333294s) [noise=true , address=145.79.227.8:4331 , piece_id=J5F7ATSE3DRAEDL6VS4OSQLE5UJSJKTB56DDQSJANKPIHCECS7QQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.055873908 (17h28m50.60734436s) [noise=true , address=145.79.227.32:4836 , piece_id=I47WTHOTKBERKTSUXEOYFKDSL22ESSBBFGHB7Z5QGHFII3NLUXUA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.056980796 (17h28m50.608451257s) [noise=true , address=145.79.227.13:4803 , piece_id=NJUXOTFWDS5NSHJTTTW444UAYUKMEJ4OA26GPJWOXFEGUW7MZN2A, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.057477022 (17h28m50.608947483s) [noise=true , address=145.79.227.9:4366 , piece_id=4YDVA4RMRUQJSR7JJZOXOYHPRCOCS75JIVSFSI7MV3UFOUY3E7OQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.058018342 (17h28m50.609488803s) [noise=true , address=145.79.227.28:4726 , piece_id=UNWVLOG6RB4BPIFB32Q5HPQ52DSHYEZ6FOYWELSDBCZB23A6AFAA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.060006197 (17h28m50.611476658s) [noise=true , address=145.79.227.21:4474 , piece_id=2W5SD2DUKNV6OQ2NK3Z4MRYMW2WTXEJ66SOQR6QK6LUOSF4XR6KA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.060567757 (17h28m50.612038218s) [noise=true , address=145.79.227.27:4691 , piece_id=5GBTJKY553CPE7EKYNIQEKVLBKUTWDBY6UZZ2TO4FQ3T3ELDE2QA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.062232697 (17h28m50.613703148s) [noise=true , address=145.79.227.8:4330 , piece_id=WO54DON3YBUKHQDWVVZUAFL3V3QCEBGEP6URNAOOFGFGNBCX33WA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.064096336 (17h28m50.615566797s) [noise=true , address=145.79.227.14:4837 , piece_id=G4RWP56E2WEPB6U6OVTPO4JS4AL2OICLX7TVTD2G3R6SATRA6GKQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.065772528 (17h28m50.617242989s) [noise=true , address=145.79.227.6:4258 , piece_id=ZPMTRI273KK7JZRSLPDMZMYTYH6VKIRMCX4IWT6GW2XNXENRKCRA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.067776333 (17h28m50.619246794s) [noise=true , address=77.90.31.12:4590 , piece_id=Y6N4IUYUMGJDFQ5EWS4X6UM7UPDFGNAZGOSV6DSVGNMYEPTPET6Q, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.069763207 (17h28m50.621233668s) [noise=true , address=145.79.227.16:4910 , piece_id=52L4KWIFZ4JWJ3DFSJZ33GAPGWMTUF22455PAJGZOQOEFCR3DOBQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.071712599 (17h28m50.62318309s) [noise=true , address=77.90.31.11:4554 , piece_id=4WL2ZCU3QIEEP4FFBOREJWIGL5QX2OHXZ2SHW4CQHW7CPCFLWSOA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:53.794020774 (17h28m52.345491235s) [upload=15416]: Next returned nil segment Nov 10 01:32:53.794034640 (17h28m52.345505101s) [upload=15416]: Waiting for error group managing segments... Nov 10 01:32:54.124111762 (17h28m52.675582223s) [noise=true , address=77.90.31.12:4590 , piece_id=Y6N4IUYUMGJDFQ5EWS4X6UM7UPDFGNAZGOSV6DSVGNMYEPTPET6Q, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:32:54.204858650 (17h28m52.756329101s) [noise=true , address=77.90.31.11:4554 , piece_id=4WL2ZCU3QIEEP4FFBOREJWIGL5QX2OHXZ2SHW4CQHW7CPCFLWSOA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:01.893062231 (17h29m0.444532702s) [noise=true , address=145.79.227.8:4331 , piece_id=J5F7ATSE3DRAEDL6VS4OSQLE5UJSJKTB56DDQSJANKPIHCECS7QQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:08.535423736 (17h29m7.086894197s) [noise=true , address=145.79.227.6:4258 , piece_id=ZPMTRI273KK7JZRSLPDMZMYTYH6VKIRMCX4IWT6GW2XNXENRKCRA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:09.147601053 (17h29m7.699071514s) [noise=true , address=145.79.227.28:4726 , piece_id=UNWVLOG6RB4BPIFB32Q5HPQ52DSHYEZ6FOYWELSDBCZB23A6AFAA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:10.101167260 (17h29m8.652637722s) [noise=true , address=145.79.227.9:4366 , piece_id=4YDVA4RMRUQJSR7JJZOXOYHPRCOCS75JIVSFSI7MV3UFOUY3E7OQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:10.453714069 (17h29m9.00518453s) [noise=true , address=145.79.227.16:4910 , piece_id=52L4KWIFZ4JWJ3DFSJZ33GAPGWMTUF22455PAJGZOQOEFCR3DOBQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:11.100100727 (17h29m9.651571188s) [noise=true , address=145.79.227.10:4403 , piece_id=CJLGVRMRLLVE2G7KDO3HA22ARYBAPQ63CKVXYG7OXH7B6U7DPMYA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:14.822230635 (17h29m13.373701096s) [noise=true , address=145.79.227.21:4474 , piece_id=2W5SD2DUKNV6OQ2NK3Z4MRYMW2WTXEJ66SOQR6QK6LUOSF4XR6KA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:17.170665030 (17h29m15.722135481s) [noise=true , address=145.79.227.27:4691 , piece_id=5GBTJKY553CPE7EKYNIQEKVLBKUTWDBY6UZZ2TO4FQ3T3ELDE2QA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:19.422554030 (17h29m17.974024491s) [noise=true , address=145.79.227.13:4803 , piece_id=NJUXOTFWDS5NSHJTTTW444UAYUKMEJ4OA26GPJWOXFEGUW7MZN2A, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:21.096313239 (17h29m19.6477837s) [noise=true , address=145.79.227.32:4836 , piece_id=I47WTHOTKBERKTSUXEOYFKDSL22ESSBBFGHB7Z5QGHFII3NLUXUA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096488773 (17h29m19.647959234s) [noise=true , address=145.79.227.8:4330 , piece_id=WO54DON3YBUKHQDWVVZUAFL3V3QCEBGEP6URNAOOFGFGNBCX33WA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096644479 (17h29m19.64811493s) [noise=true , address=145.79.227.13:4802 , piece_id=FQ6JKNKMAMV5PVGMA5XTX7AVMJNL4OCF353IJQGRU72KT3UIK7FA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096788984 (17h29m19.648259445s) [noise=true , address=145.79.227.25:4618 , piece_id=WGVJKWE4FJO6QFKPXMZ4QUYY5H6FLKXLXHU7N2VCIPJI5GXWBN5Q, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096954390 (17h29m19.648424861s) [noise=true , address=145.79.227.14:4837 , piece_id=G4RWP56E2WEPB6U6OVTPO4JS4AL2OICLX7TVTD2G3R6SATRA6GKQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection Nov 10 01:33:21.096989987 (17h29m19.648460448s) [seg_pos={0 0}, upload=15416]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:33:21.342594608 (17h29m19.894065069s) [upload=15416]: Done uploading segments... Nov 10 01:33:21.342640315 (17h29m19.894110766s) [upload=15416]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.747335469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X...X......] duration: 5.620067814s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:34:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.620088984s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 54.661726ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6c987828a72ec3e8 FAIL (2025-11-10 01:34:10 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:34562->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 54.692675ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X......................X....X.....................X...X..X..] duration: 16.705121349s trace: https://jaeger.dev.storj.io/trace/51b7178e105d4206 FAIL (2025-11-10 01:34:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:32:51.801397615 (17h28m50.352868076s) [upload=15416]: Starting upload Nov 10 01:32:51.801485843 (17h28m50.352956305s) [upload=15416]: Done starting upload Nov 10 01:32:51.801680303 (17h28m50.353150764s) [upload=15416]: Uploading segments... Nov 10 01:32:51.801845388 (17h28m50.353315839s) [upload=15416]: Got next segment. Inline: false Nov 10 01:32:51.801927615 (17h28m50.353398076s) [upload=15416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:32:51.801935770 (17h28m50.353406231s) [upload=15416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:32:52.053537580 (17h28m50.605008041s) [seg_pos={0 0}, upload=15416]: Begin upload segment... Nov 10 01:32:52.053623583 (17h28m50.605094044s) [seg_pos={0 0}, upload=15416]: Done begin upload segment. Nov 10 01:32:52.053688186 (17h28m50.605158637s) [noise=true , address=145.79.227.13:4802 , piece_id=FQ6JKNKMAMV5PVGMA5XTX7AVMJNL4OCF353IJQGRU72KT3UIK7FA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.053715909 (17h28m50.60518637s) [noise=true , address=145.79.227.10:4403 , piece_id=CJLGVRMRLLVE2G7KDO3HA22ARYBAPQ63CKVXYG7OXH7B6U7DPMYA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.054433696 (17h28m50.605904158s) [noise=true , address=145.79.227.25:4618 , piece_id=WGVJKWE4FJO6QFKPXMZ4QUYY5H6FLKXLXHU7N2VCIPJI5GXWBN5Q, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.054862843 (17h28m50.606333294s) [noise=true , address=145.79.227.8:4331 , piece_id=J5F7ATSE3DRAEDL6VS4OSQLE5UJSJKTB56DDQSJANKPIHCECS7QQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.055873908 (17h28m50.60734436s) [noise=true , address=145.79.227.32:4836 , piece_id=I47WTHOTKBERKTSUXEOYFKDSL22ESSBBFGHB7Z5QGHFII3NLUXUA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.056980796 (17h28m50.608451257s) [noise=true , address=145.79.227.13:4803 , piece_id=NJUXOTFWDS5NSHJTTTW444UAYUKMEJ4OA26GPJWOXFEGUW7MZN2A, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.057477022 (17h28m50.608947483s) [noise=true , address=145.79.227.9:4366 , piece_id=4YDVA4RMRUQJSR7JJZOXOYHPRCOCS75JIVSFSI7MV3UFOUY3E7OQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.058018342 (17h28m50.609488803s) [noise=true , address=145.79.227.28:4726 , piece_id=UNWVLOG6RB4BPIFB32Q5HPQ52DSHYEZ6FOYWELSDBCZB23A6AFAA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.060006197 (17h28m50.611476658s) [noise=true , address=145.79.227.21:4474 , piece_id=2W5SD2DUKNV6OQ2NK3Z4MRYMW2WTXEJ66SOQR6QK6LUOSF4XR6KA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.060567757 (17h28m50.612038218s) [noise=true , address=145.79.227.27:4691 , piece_id=5GBTJKY553CPE7EKYNIQEKVLBKUTWDBY6UZZ2TO4FQ3T3ELDE2QA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.062232697 (17h28m50.613703148s) [noise=true , address=145.79.227.8:4330 , piece_id=WO54DON3YBUKHQDWVVZUAFL3V3QCEBGEP6URNAOOFGFGNBCX33WA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.064096336 (17h28m50.615566797s) [noise=true , address=145.79.227.14:4837 , piece_id=G4RWP56E2WEPB6U6OVTPO4JS4AL2OICLX7TVTD2G3R6SATRA6GKQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.065772528 (17h28m50.617242989s) [noise=true , address=145.79.227.6:4258 , piece_id=ZPMTRI273KK7JZRSLPDMZMYTYH6VKIRMCX4IWT6GW2XNXENRKCRA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.067776333 (17h28m50.619246794s) [noise=true , address=77.90.31.12:4590 , piece_id=Y6N4IUYUMGJDFQ5EWS4X6UM7UPDFGNAZGOSV6DSVGNMYEPTPET6Q, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.069763207 (17h28m50.621233668s) [noise=true , address=145.79.227.16:4910 , piece_id=52L4KWIFZ4JWJ3DFSJZ33GAPGWMTUF22455PAJGZOQOEFCR3DOBQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.071712599 (17h28m50.62318309s) [noise=true , address=77.90.31.11:4554 , piece_id=4WL2ZCU3QIEEP4FFBOREJWIGL5QX2OHXZ2SHW4CQHW7CPCFLWSOA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:53.794020774 (17h28m52.345491235s) [upload=15416]: Next returned nil segment Nov 10 01:32:53.794034640 (17h28m52.345505101s) [upload=15416]: Waiting for error group managing segments... Nov 10 01:32:54.124111762 (17h28m52.675582223s) [noise=true , address=77.90.31.12:4590 , piece_id=Y6N4IUYUMGJDFQ5EWS4X6UM7UPDFGNAZGOSV6DSVGNMYEPTPET6Q, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:32:54.204858650 (17h28m52.756329101s) [noise=true , address=77.90.31.11:4554 , piece_id=4WL2ZCU3QIEEP4FFBOREJWIGL5QX2OHXZ2SHW4CQHW7CPCFLWSOA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:01.893062231 (17h29m0.444532702s) [noise=true , address=145.79.227.8:4331 , piece_id=J5F7ATSE3DRAEDL6VS4OSQLE5UJSJKTB56DDQSJANKPIHCECS7QQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:08.535423736 (17h29m7.086894197s) [noise=true , address=145.79.227.6:4258 , piece_id=ZPMTRI273KK7JZRSLPDMZMYTYH6VKIRMCX4IWT6GW2XNXENRKCRA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:09.147601053 (17h29m7.699071514s) [noise=true , address=145.79.227.28:4726 , piece_id=UNWVLOG6RB4BPIFB32Q5HPQ52DSHYEZ6FOYWELSDBCZB23A6AFAA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:10.101167260 (17h29m8.652637722s) [noise=true , address=145.79.227.9:4366 , piece_id=4YDVA4RMRUQJSR7JJZOXOYHPRCOCS75JIVSFSI7MV3UFOUY3E7OQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:10.453714069 (17h29m9.00518453s) [noise=true , address=145.79.227.16:4910 , piece_id=52L4KWIFZ4JWJ3DFSJZ33GAPGWMTUF22455PAJGZOQOEFCR3DOBQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:11.100100727 (17h29m9.651571188s) [noise=true , address=145.79.227.10:4403 , piece_id=CJLGVRMRLLVE2G7KDO3HA22ARYBAPQ63CKVXYG7OXH7B6U7DPMYA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:14.822230635 (17h29m13.373701096s) [noise=true , address=145.79.227.21:4474 , piece_id=2W5SD2DUKNV6OQ2NK3Z4MRYMW2WTXEJ66SOQR6QK6LUOSF4XR6KA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:17.170665030 (17h29m15.722135481s) [noise=true , address=145.79.227.27:4691 , piece_id=5GBTJKY553CPE7EKYNIQEKVLBKUTWDBY6UZZ2TO4FQ3T3ELDE2QA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:19.422554030 (17h29m17.974024491s) [noise=true , address=145.79.227.13:4803 , piece_id=NJUXOTFWDS5NSHJTTTW444UAYUKMEJ4OA26GPJWOXFEGUW7MZN2A, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:21.096313239 (17h29m19.6477837s) [noise=true , address=145.79.227.32:4836 , piece_id=I47WTHOTKBERKTSUXEOYFKDSL22ESSBBFGHB7Z5QGHFII3NLUXUA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096488773 (17h29m19.647959234s) [noise=true , address=145.79.227.8:4330 , piece_id=WO54DON3YBUKHQDWVVZUAFL3V3QCEBGEP6URNAOOFGFGNBCX33WA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096644479 (17h29m19.64811493s) [noise=true , address=145.79.227.13:4802 , piece_id=FQ6JKNKMAMV5PVGMA5XTX7AVMJNL4OCF353IJQGRU72KT3UIK7FA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096788984 (17h29m19.648259445s) [noise=true , address=145.79.227.25:4618 , piece_id=WGVJKWE4FJO6QFKPXMZ4QUYY5H6FLKXLXHU7N2VCIPJI5GXWBN5Q, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096954390 (17h29m19.648424861s) [noise=true , address=145.79.227.14:4837 , piece_id=G4RWP56E2WEPB6U6OVTPO4JS4AL2OICLX7TVTD2G3R6SATRA6GKQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection Nov 10 01:33:21.096989987 (17h29m19.648460448s) [seg_pos={0 0}, upload=15416]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:33:21.342594608 (17h29m19.894065069s) [upload=15416]: Done uploading segments... Nov 10 01:33:21.342640315 (17h29m19.894110766s) [upload=15416]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 16.705149614s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 61.527831ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6c987828a72ec3e8 FAIL (2025-11-10 01:33:50 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:34562->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 61.556025ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X...X.....] duration: 7.752548163s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:33:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.752620922s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X......................X....X.....................X...X..X.] duration: 20.199722942s trace: https://jaeger.dev.storj.io/trace/51b7178e105d4206 FAIL (2025-11-10 01:33:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:32:51.801397615 (17h28m50.352868076s) [upload=15416]: Starting upload Nov 10 01:32:51.801485843 (17h28m50.352956305s) [upload=15416]: Done starting upload Nov 10 01:32:51.801680303 (17h28m50.353150764s) [upload=15416]: Uploading segments... Nov 10 01:32:51.801845388 (17h28m50.353315839s) [upload=15416]: Got next segment. Inline: false Nov 10 01:32:51.801927615 (17h28m50.353398076s) [upload=15416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:32:51.801935770 (17h28m50.353406231s) [upload=15416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:32:52.053537580 (17h28m50.605008041s) [seg_pos={0 0}, upload=15416]: Begin upload segment... Nov 10 01:32:52.053623583 (17h28m50.605094044s) [seg_pos={0 0}, upload=15416]: Done begin upload segment. Nov 10 01:32:52.053688186 (17h28m50.605158637s) [noise=true , address=145.79.227.13:4802 , piece_id=FQ6JKNKMAMV5PVGMA5XTX7AVMJNL4OCF353IJQGRU72KT3UIK7FA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.053715909 (17h28m50.60518637s) [noise=true , address=145.79.227.10:4403 , piece_id=CJLGVRMRLLVE2G7KDO3HA22ARYBAPQ63CKVXYG7OXH7B6U7DPMYA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.054433696 (17h28m50.605904158s) [noise=true , address=145.79.227.25:4618 , piece_id=WGVJKWE4FJO6QFKPXMZ4QUYY5H6FLKXLXHU7N2VCIPJI5GXWBN5Q, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.054862843 (17h28m50.606333294s) [noise=true , address=145.79.227.8:4331 , piece_id=J5F7ATSE3DRAEDL6VS4OSQLE5UJSJKTB56DDQSJANKPIHCECS7QQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.055873908 (17h28m50.60734436s) [noise=true , address=145.79.227.32:4836 , piece_id=I47WTHOTKBERKTSUXEOYFKDSL22ESSBBFGHB7Z5QGHFII3NLUXUA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.056980796 (17h28m50.608451257s) [noise=true , address=145.79.227.13:4803 , piece_id=NJUXOTFWDS5NSHJTTTW444UAYUKMEJ4OA26GPJWOXFEGUW7MZN2A, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.057477022 (17h28m50.608947483s) [noise=true , address=145.79.227.9:4366 , piece_id=4YDVA4RMRUQJSR7JJZOXOYHPRCOCS75JIVSFSI7MV3UFOUY3E7OQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.058018342 (17h28m50.609488803s) [noise=true , address=145.79.227.28:4726 , piece_id=UNWVLOG6RB4BPIFB32Q5HPQ52DSHYEZ6FOYWELSDBCZB23A6AFAA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.060006197 (17h28m50.611476658s) [noise=true , address=145.79.227.21:4474 , piece_id=2W5SD2DUKNV6OQ2NK3Z4MRYMW2WTXEJ66SOQR6QK6LUOSF4XR6KA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.060567757 (17h28m50.612038218s) [noise=true , address=145.79.227.27:4691 , piece_id=5GBTJKY553CPE7EKYNIQEKVLBKUTWDBY6UZZ2TO4FQ3T3ELDE2QA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.062232697 (17h28m50.613703148s) [noise=true , address=145.79.227.8:4330 , piece_id=WO54DON3YBUKHQDWVVZUAFL3V3QCEBGEP6URNAOOFGFGNBCX33WA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.064096336 (17h28m50.615566797s) [noise=true , address=145.79.227.14:4837 , piece_id=G4RWP56E2WEPB6U6OVTPO4JS4AL2OICLX7TVTD2G3R6SATRA6GKQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.065772528 (17h28m50.617242989s) [noise=true , address=145.79.227.6:4258 , piece_id=ZPMTRI273KK7JZRSLPDMZMYTYH6VKIRMCX4IWT6GW2XNXENRKCRA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.067776333 (17h28m50.619246794s) [noise=true , address=77.90.31.12:4590 , piece_id=Y6N4IUYUMGJDFQ5EWS4X6UM7UPDFGNAZGOSV6DSVGNMYEPTPET6Q, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.069763207 (17h28m50.621233668s) [noise=true , address=145.79.227.16:4910 , piece_id=52L4KWIFZ4JWJ3DFSJZ33GAPGWMTUF22455PAJGZOQOEFCR3DOBQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.071712599 (17h28m50.62318309s) [noise=true , address=77.90.31.11:4554 , piece_id=4WL2ZCU3QIEEP4FFBOREJWIGL5QX2OHXZ2SHW4CQHW7CPCFLWSOA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:53.794020774 (17h28m52.345491235s) [upload=15416]: Next returned nil segment Nov 10 01:32:53.794034640 (17h28m52.345505101s) [upload=15416]: Waiting for error group managing segments... Nov 10 01:32:54.124111762 (17h28m52.675582223s) [noise=true , address=77.90.31.12:4590 , piece_id=Y6N4IUYUMGJDFQ5EWS4X6UM7UPDFGNAZGOSV6DSVGNMYEPTPET6Q, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:32:54.204858650 (17h28m52.756329101s) [noise=true , address=77.90.31.11:4554 , piece_id=4WL2ZCU3QIEEP4FFBOREJWIGL5QX2OHXZ2SHW4CQHW7CPCFLWSOA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:01.893062231 (17h29m0.444532702s) [noise=true , address=145.79.227.8:4331 , piece_id=J5F7ATSE3DRAEDL6VS4OSQLE5UJSJKTB56DDQSJANKPIHCECS7QQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:08.535423736 (17h29m7.086894197s) [noise=true , address=145.79.227.6:4258 , piece_id=ZPMTRI273KK7JZRSLPDMZMYTYH6VKIRMCX4IWT6GW2XNXENRKCRA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:09.147601053 (17h29m7.699071514s) [noise=true , address=145.79.227.28:4726 , piece_id=UNWVLOG6RB4BPIFB32Q5HPQ52DSHYEZ6FOYWELSDBCZB23A6AFAA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:10.101167260 (17h29m8.652637722s) [noise=true , address=145.79.227.9:4366 , piece_id=4YDVA4RMRUQJSR7JJZOXOYHPRCOCS75JIVSFSI7MV3UFOUY3E7OQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:10.453714069 (17h29m9.00518453s) [noise=true , address=145.79.227.16:4910 , piece_id=52L4KWIFZ4JWJ3DFSJZ33GAPGWMTUF22455PAJGZOQOEFCR3DOBQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:11.100100727 (17h29m9.651571188s) [noise=true , address=145.79.227.10:4403 , piece_id=CJLGVRMRLLVE2G7KDO3HA22ARYBAPQ63CKVXYG7OXH7B6U7DPMYA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:14.822230635 (17h29m13.373701096s) [noise=true , address=145.79.227.21:4474 , piece_id=2W5SD2DUKNV6OQ2NK3Z4MRYMW2WTXEJ66SOQR6QK6LUOSF4XR6KA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:17.170665030 (17h29m15.722135481s) [noise=true , address=145.79.227.27:4691 , piece_id=5GBTJKY553CPE7EKYNIQEKVLBKUTWDBY6UZZ2TO4FQ3T3ELDE2QA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:19.422554030 (17h29m17.974024491s) [noise=true , address=145.79.227.13:4803 , piece_id=NJUXOTFWDS5NSHJTTTW444UAYUKMEJ4OA26GPJWOXFEGUW7MZN2A, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:21.096313239 (17h29m19.6477837s) [noise=true , address=145.79.227.32:4836 , piece_id=I47WTHOTKBERKTSUXEOYFKDSL22ESSBBFGHB7Z5QGHFII3NLUXUA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096488773 (17h29m19.647959234s) [noise=true , address=145.79.227.8:4330 , piece_id=WO54DON3YBUKHQDWVVZUAFL3V3QCEBGEP6URNAOOFGFGNBCX33WA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096644479 (17h29m19.64811493s) [noise=true , address=145.79.227.13:4802 , piece_id=FQ6JKNKMAMV5PVGMA5XTX7AVMJNL4OCF353IJQGRU72KT3UIK7FA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096788984 (17h29m19.648259445s) [noise=true , address=145.79.227.25:4618 , piece_id=WGVJKWE4FJO6QFKPXMZ4QUYY5H6FLKXLXHU7N2VCIPJI5GXWBN5Q, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096954390 (17h29m19.648424861s) [noise=true , address=145.79.227.14:4837 , piece_id=G4RWP56E2WEPB6U6OVTPO4JS4AL2OICLX7TVTD2G3R6SATRA6GKQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection Nov 10 01:33:21.096989987 (17h29m19.648460448s) [seg_pos={0 0}, upload=15416]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:33:21.342594608 (17h29m19.894065069s) [upload=15416]: Done uploading segments... Nov 10 01:33:21.342640315 (17h29m19.894110766s) [upload=15416]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 20.199743902s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 266.861566ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6c987828a72ec3e8 FAIL (2025-11-10 01:33:29 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:34562->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 266.914798ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X......................X....X.....................X...X..X] duration: 30.248416079s trace: https://jaeger.dev.storj.io/trace/51b7178e105d4206 FAIL (2025-11-10 01:33:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:32:51.801397615 (17h28m50.352868076s) [upload=15416]: Starting upload Nov 10 01:32:51.801485843 (17h28m50.352956305s) [upload=15416]: Done starting upload Nov 10 01:32:51.801680303 (17h28m50.353150764s) [upload=15416]: Uploading segments... Nov 10 01:32:51.801845388 (17h28m50.353315839s) [upload=15416]: Got next segment. Inline: false Nov 10 01:32:51.801927615 (17h28m50.353398076s) [upload=15416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:32:51.801935770 (17h28m50.353406231s) [upload=15416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:32:52.053537580 (17h28m50.605008041s) [seg_pos={0 0}, upload=15416]: Begin upload segment... Nov 10 01:32:52.053623583 (17h28m50.605094044s) [seg_pos={0 0}, upload=15416]: Done begin upload segment. Nov 10 01:32:52.053688186 (17h28m50.605158637s) [noise=true , address=145.79.227.13:4802 , piece_id=FQ6JKNKMAMV5PVGMA5XTX7AVMJNL4OCF353IJQGRU72KT3UIK7FA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.053715909 (17h28m50.60518637s) [noise=true , address=145.79.227.10:4403 , piece_id=CJLGVRMRLLVE2G7KDO3HA22ARYBAPQ63CKVXYG7OXH7B6U7DPMYA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.054433696 (17h28m50.605904158s) [noise=true , address=145.79.227.25:4618 , piece_id=WGVJKWE4FJO6QFKPXMZ4QUYY5H6FLKXLXHU7N2VCIPJI5GXWBN5Q, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.054862843 (17h28m50.606333294s) [noise=true , address=145.79.227.8:4331 , piece_id=J5F7ATSE3DRAEDL6VS4OSQLE5UJSJKTB56DDQSJANKPIHCECS7QQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.055873908 (17h28m50.60734436s) [noise=true , address=145.79.227.32:4836 , piece_id=I47WTHOTKBERKTSUXEOYFKDSL22ESSBBFGHB7Z5QGHFII3NLUXUA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.056980796 (17h28m50.608451257s) [noise=true , address=145.79.227.13:4803 , piece_id=NJUXOTFWDS5NSHJTTTW444UAYUKMEJ4OA26GPJWOXFEGUW7MZN2A, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.057477022 (17h28m50.608947483s) [noise=true , address=145.79.227.9:4366 , piece_id=4YDVA4RMRUQJSR7JJZOXOYHPRCOCS75JIVSFSI7MV3UFOUY3E7OQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.058018342 (17h28m50.609488803s) [noise=true , address=145.79.227.28:4726 , piece_id=UNWVLOG6RB4BPIFB32Q5HPQ52DSHYEZ6FOYWELSDBCZB23A6AFAA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.060006197 (17h28m50.611476658s) [noise=true , address=145.79.227.21:4474 , piece_id=2W5SD2DUKNV6OQ2NK3Z4MRYMW2WTXEJ66SOQR6QK6LUOSF4XR6KA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.060567757 (17h28m50.612038218s) [noise=true , address=145.79.227.27:4691 , piece_id=5GBTJKY553CPE7EKYNIQEKVLBKUTWDBY6UZZ2TO4FQ3T3ELDE2QA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.062232697 (17h28m50.613703148s) [noise=true , address=145.79.227.8:4330 , piece_id=WO54DON3YBUKHQDWVVZUAFL3V3QCEBGEP6URNAOOFGFGNBCX33WA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.064096336 (17h28m50.615566797s) [noise=true , address=145.79.227.14:4837 , piece_id=G4RWP56E2WEPB6U6OVTPO4JS4AL2OICLX7TVTD2G3R6SATRA6GKQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.065772528 (17h28m50.617242989s) [noise=true , address=145.79.227.6:4258 , piece_id=ZPMTRI273KK7JZRSLPDMZMYTYH6VKIRMCX4IWT6GW2XNXENRKCRA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.067776333 (17h28m50.619246794s) [noise=true , address=77.90.31.12:4590 , piece_id=Y6N4IUYUMGJDFQ5EWS4X6UM7UPDFGNAZGOSV6DSVGNMYEPTPET6Q, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.069763207 (17h28m50.621233668s) [noise=true , address=145.79.227.16:4910 , piece_id=52L4KWIFZ4JWJ3DFSJZ33GAPGWMTUF22455PAJGZOQOEFCR3DOBQ, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:52.071712599 (17h28m50.62318309s) [noise=true , address=77.90.31.11:4554 , piece_id=4WL2ZCU3QIEEP4FFBOREJWIGL5QX2OHXZ2SHW4CQHW7CPCFLWSOA, seg_pos={0 0}, upload=15416]: Uploading piece... Nov 10 01:32:53.794020774 (17h28m52.345491235s) [upload=15416]: Next returned nil segment Nov 10 01:32:53.794034640 (17h28m52.345505101s) [upload=15416]: Waiting for error group managing segments... Nov 10 01:32:54.124111762 (17h28m52.675582223s) [noise=true , address=77.90.31.12:4590 , piece_id=Y6N4IUYUMGJDFQ5EWS4X6UM7UPDFGNAZGOSV6DSVGNMYEPTPET6Q, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:32:54.204858650 (17h28m52.756329101s) [noise=true , address=77.90.31.11:4554 , piece_id=4WL2ZCU3QIEEP4FFBOREJWIGL5QX2OHXZ2SHW4CQHW7CPCFLWSOA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:01.893062231 (17h29m0.444532702s) [noise=true , address=145.79.227.8:4331 , piece_id=J5F7ATSE3DRAEDL6VS4OSQLE5UJSJKTB56DDQSJANKPIHCECS7QQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:08.535423736 (17h29m7.086894197s) [noise=true , address=145.79.227.6:4258 , piece_id=ZPMTRI273KK7JZRSLPDMZMYTYH6VKIRMCX4IWT6GW2XNXENRKCRA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:09.147601053 (17h29m7.699071514s) [noise=true , address=145.79.227.28:4726 , piece_id=UNWVLOG6RB4BPIFB32Q5HPQ52DSHYEZ6FOYWELSDBCZB23A6AFAA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:10.101167260 (17h29m8.652637722s) [noise=true , address=145.79.227.9:4366 , piece_id=4YDVA4RMRUQJSR7JJZOXOYHPRCOCS75JIVSFSI7MV3UFOUY3E7OQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:10.453714069 (17h29m9.00518453s) [noise=true , address=145.79.227.16:4910 , piece_id=52L4KWIFZ4JWJ3DFSJZ33GAPGWMTUF22455PAJGZOQOEFCR3DOBQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:11.100100727 (17h29m9.651571188s) [noise=true , address=145.79.227.10:4403 , piece_id=CJLGVRMRLLVE2G7KDO3HA22ARYBAPQ63CKVXYG7OXH7B6U7DPMYA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:14.822230635 (17h29m13.373701096s) [noise=true , address=145.79.227.21:4474 , piece_id=2W5SD2DUKNV6OQ2NK3Z4MRYMW2WTXEJ66SOQR6QK6LUOSF4XR6KA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:17.170665030 (17h29m15.722135481s) [noise=true , address=145.79.227.27:4691 , piece_id=5GBTJKY553CPE7EKYNIQEKVLBKUTWDBY6UZZ2TO4FQ3T3ELDE2QA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:19.422554030 (17h29m17.974024491s) [noise=true , address=145.79.227.13:4803 , piece_id=NJUXOTFWDS5NSHJTTTW444UAYUKMEJ4OA26GPJWOXFEGUW7MZN2A, seg_pos={0 0}, upload=15416]: Done uploading piece. err: <nil> Nov 10 01:33:21.096313239 (17h29m19.6477837s) [noise=true , address=145.79.227.32:4836 , piece_id=I47WTHOTKBERKTSUXEOYFKDSL22ESSBBFGHB7Z5QGHFII3NLUXUA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096488773 (17h29m19.647959234s) [noise=true , address=145.79.227.8:4330 , piece_id=WO54DON3YBUKHQDWVVZUAFL3V3QCEBGEP6URNAOOFGFGNBCX33WA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096644479 (17h29m19.64811493s) [noise=true , address=145.79.227.13:4802 , piece_id=FQ6JKNKMAMV5PVGMA5XTX7AVMJNL4OCF353IJQGRU72KT3UIK7FA, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096788984 (17h29m19.648259445s) [noise=true , address=145.79.227.25:4618 , piece_id=WGVJKWE4FJO6QFKPXMZ4QUYY5H6FLKXLXHU7N2VCIPJI5GXWBN5Q, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:33:21.096954390 (17h29m19.648424861s) [noise=true , address=145.79.227.14:4837 , piece_id=G4RWP56E2WEPB6U6OVTPO4JS4AL2OICLX7TVTD2G3R6SATRA6GKQ, seg_pos={0 0}, upload=15416]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47924->145.79.227.14:4837: use of closed network connection Nov 10 01:33:21.096989987 (17h29m19.648460448s) [seg_pos={0 0}, upload=15416]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:33:21.342594608 (17h29m19.894065069s) [upload=15416]: Done uploading segments... Nov 10 01:33:21.342640315 (17h29m19.894110766s) [upload=15416]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.248446697s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X...X....] duration: 19.363220089s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:32:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.363249174s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [X..X...........] duration: 63.23315ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6c987828a72ec3e8 FAIL (2025-11-10 01:32:50 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:34562->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 63.261584ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X...X...] duration: 5.65006186s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:32:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.650108508s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [.X..X..........] duration: 55.059542ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6c987828a72ec3e8 FAIL (2025-11-10 01:32:12 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:34562->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 55.089701ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X......................X....X.....................X...X..] duration: 5.962916156s trace: https://jaeger.dev.storj.io/trace/5cf76b08f1181d7d FAIL (2025-11-10 01:31:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 0}, upload=15376]: Uploading piece... Nov 10 01:29:51.343073484 (17h25m49.894543955s) [noise=true , address=145.79.227.25:4618 , piece_id=APVJZGBVMJAETROMHFYBYMNIMGOYH3WNEUGDB4IZYOQYVEVEBBYA, seg_pos={0 0}, upload=15376]: Uploading piece... Nov 10 01:29:51.344185972 (17h25m49.895656433s) [noise=true , address=145.79.227.16:4909 , piece_id=WTPJAQJOBMYMQT3GIRDJ2QU735LDZ4PNI7J6GQSR4LY4KUQOLI5Q, seg_pos={0 0}, upload=15376]: Uploading piece... Nov 10 01:29:51.344622414 (17h25m49.896092875s) [noise=true , address=145.79.227.31:4798 , piece_id=BIVGYKUYNDOIB4ZUM4LHRVFUCCGITZLLX452EUY3PBYP347S5ITQ, seg_pos={0 0}, upload=15376]: Uploading piece... Nov 10 01:29:51.345560460 (17h25m49.897030921s) [noise=true , address=145.79.227.32:4836 , piece_id=56MK633HDC6E76TKDOE2FDBHXZL2RAIIMGVR5LM5ZPY3KEEEVEPQ, seg_pos={0 0}, upload=15376]: Uploading piece... Nov 10 01:29:51.346084508 (17h25m49.897554969s) [noise=true , address=145.79.227.14:4840 , piece_id=LIVSFVZWTZKAK6QEHIYMMTOUAAHIRYG4DHW5DZPZRJ3TLBTQ42OQ, seg_pos={0 0}, upload=15376]: Uploading piece... Nov 10 01:29:51.346240064 (17h25m49.897710535s) [noise=true , address=145.79.227.11:4731 , piece_id=YXM4AB64Q6QXZIZDEG7MXU3JDJMH734EXP7DCKJ5FDA4RBNVCF5A, seg_pos={0 0}, upload=15376]: Uploading piece... Nov 10 01:29:51.347343625 (17h25m49.898814087s) [noise=true , address=77.90.31.19:4842 , piece_id=27HUTX66YIYFRFN2NH4FTDTK4NXNV6ALFR4NMZFITUZMSQ3VFILA, seg_pos={0 0}, upload=15376]: Uploading piece... Nov 10 01:29:51.347462702 (17h25m49.898933163s) [noise=true , address=145.79.227.32:4835 , piece_id=UEWWLNHLZ2WVU3QHQLDTJFJJUQCH6QHCNREQGLNIOTYAP5NVZTSA, seg_pos={0 0}, upload=15376]: Uploading piece... Nov 10 01:29:51.347821936 (17h25m49.899292398s) [noise=true , address=145.79.227.31:4797 , piece_id=BBLLKYR5N6CIN6YNJ5KAM4F3IZHQNKR46GOS7J5H4GK3Y5CSPWQA, seg_pos={0 0}, upload=15376]: Uploading piece... Nov 10 01:29:51.347996178 (17h25m49.899466639s) [noise=true , address=145.79.227.6:4258 , piece_id=TL25QPCECYJZCXHFJ6LBXHIIZNQDDTBMAA3BQID4FA2MKQQ7UFYA, seg_pos={0 0}, upload=15376]: Uploading piece... Nov 10 01:29:53.102627611 (17h25m51.654098073s) [upload=15376]: Next returned nil segment Nov 10 01:29:53.102644193 (17h25m51.654114654s) [upload=15376]: Waiting for error group managing segments... Nov 10 01:29:53.449068933 (17h25m52.000539394s) [noise=true , address=77.90.31.11:4555 , piece_id=VEYRO6SMP2FHRPRI2NBKVDI6QX3CAE7FWKBUL2OJTGTFPD2PL35Q, seg_pos={0 0}, upload=15376]: Done uploading piece. err: <nil> Nov 10 01:29:53.460321581 (17h25m52.011792032s) [noise=true , address=77.90.31.19:4842 , piece_id=27HUTX66YIYFRFN2NH4FTDTK4NXNV6ALFR4NMZFITUZMSQ3VFILA, seg_pos={0 0}, upload=15376]: Done uploading piece. err: <nil> Nov 10 01:29:53.477441290 (17h25m52.028911751s) [noise=true , address=77.90.31.19:4843 , piece_id=CZEACMC6IW55NA42HSIUYMCKY6PBDLGR52PZCQPW6YVTGLH732YQ, seg_pos={0 0}, upload=15376]: Done uploading piece. err: <nil> Nov 10 01:29:53.537031722 (17h25m52.088502183s) [noise=true , address=77.90.31.16:4735 , piece_id=H7Q7SJ5VYPWEU3O7B3H65EVHHVA4TSMGUHYHNJRRWZOA654654UA, seg_pos={0 0}, upload=15376]: Done uploading piece. err: <nil> Nov 10 01:30:20.349426975 (17h26m18.900897437s) [noise=true , address=145.79.227.10:4401 , piece_id=BVN5USYPDD4SYTXVH73SPE7TIPBITXPONK3OAQK3Z56BSPCKM7TA, seg_pos={0 0}, upload=15376]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:52112->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:52112->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52112->145.79.227.10:4401: use of closed network connection Nov 10 01:30:20.349628759 (17h26m18.90109922s) [noise=true , address=145.79.227.8:4331 , piece_id=D5ITSPFR3RYGBIYHQGCE347AXPK6RHNTARO2CZTVUDROARDEO7CQ, seg_pos={0 0}, upload=15376]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:35598->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:35598->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35598->145.79.227.8:4331: use of closed network connection Nov 10 01:30:20.349843138 (17h26m18.901313599s) [noise=true , address=145.79.227.14:4840 , piece_id=LIVSFVZWTZKAK6QEHIYMMTOUAAHIRYG4DHW5DZPZRJ3TLBTQ42OQ, seg_pos={0 0}, upload=15376]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:35114->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:20.349993233 (17h26m18.901463725s) [noise=true , address=145.79.227.31:4798 , piece_id=BIVGYKUYNDOIB4ZUM4LHRVFUCCGITZLLX452EUY3PBYP347S5ITQ, seg_pos={0 0}, upload=15376]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:49418->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:49418->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49418->145.79.227.31:4798: use of closed network connection Nov 10 01:30:20.350255422 (17h26m18.901725883s) [noise=true , address=145.79.227.11:4731 , piece_id=YXM4AB64Q6QXZIZDEG7MXU3JDJMH734EXP7DCKJ5FDA4RBNVCF5A, seg_pos={0 0}, upload=15376]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:48082->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:48082->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48082->145.79.227.11:4731: use of closed network connection Nov 10 01:30:20.350396310 (17h26m18.901866761s) [noise=true , address=145.79.227.31:4797 , piece_id=BBLLKYR5N6CIN6YNJ5KAM4F3IZHQNKR46GOS7J5H4GK3Y5CSPWQA, seg_pos={0 0}, upload=15376]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: read tcp 188.166.225.170:44116->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:20.350605639 (17h26m18.90207613s) [noise=true , address=145.79.227.6:4258 , piece_id=TL25QPCECYJZCXHFJ6LBXHIIZNQDDTBMAA3BQID4FA2MKQQ7UFYA, seg_pos={0 0}, upload=15376]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:39984->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:39984->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39984->145.79.227.6:4258: use of closed network connection Nov 10 01:30:20.350810499 (17h26m18.90228096s) [noise=true , address=145.79.227.26:4653 , piece_id=MJEZU6AGZ2A27MQEZRODKX6FZ5JUSJAFGX2RTGRMILUXQUW4N5DA, seg_pos={0 0}, upload=15376]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: read tcp 188.166.225.170:39946->145.79.227.26:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:20.350977307 (17h26m18.902447758s) [noise=true , address=145.79.227.25:4618 , piece_id=APVJZGBVMJAETROMHFYBYMNIMGOYH3WNEUGDB4IZYOQYVEVEBBYA, seg_pos={0 0}, upload=15376]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:48396->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:20.350999710 (17h26m18.902470171s) [noise=true , address=145.79.227.32:4836 , piece_id=56MK633HDC6E76TKDOE2FDBHXZL2RAIIMGVR5LM5ZPY3KEEEVEPQ, seg_pos={0 0}, upload=15376]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:43862->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:20.351185924 (17h26m18.902656385s) [noise=true , address=145.79.227.16:4909 , piece_id=WTPJAQJOBMYMQT3GIRDJ2QU735LDZ4PNI7J6GQSR4LY4KUQOLI5Q, seg_pos={0 0}, upload=15376]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:20.351257801 (17h26m18.902728262s) [noise=true , address=145.79.227.32:4835 , piece_id=UEWWLNHLZ2WVU3QHQLDTJFJJUQCH6QHCNREQGLNIOTYAP5NVZTSA, seg_pos={0 0}, upload=15376]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: read tcp 188.166.225.170:59996->145.79.227.32:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:20.351378481 (17h26m18.902848932s) [seg_pos={0 0}, upload=15376]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 10 01:30:20.608534016 (17h26m19.160004477s) [upload=15376]: Done uploading segments... Nov 10 01:30:20.608587518 (17h26m19.160057979s) [upload=15376]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.962950571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration: 5.63596146s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:31:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.635991386s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [..X..X.........] duration: 67.883394ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6c987828a72ec3e8 FAIL (2025-11-10 01:31:28 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:34562->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 67.907248ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [...X..X........] duration: 47.730447ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/6c987828a72ec3e8 FAIL (2025-11-10 01:30:56 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:34562->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 47.782136ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X...X.] duration: 6.896565095s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:30:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.896613847s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)