Orbiter History

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 73.233461ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 01:58:37 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 73.259158ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X...X....................................] duration: 5.690502663s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:58:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.690531148s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 69.897357ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 01:58:12 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 69.924539ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X...X..X......XX........................] duration: 5.900839202s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:57:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.900869479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X...X...................................] duration: 5.641657461s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:57:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.641686306s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...X..................................] duration: 5.654536455s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:57:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.654570199s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 64.839397ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 01:57:11 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 64.865696ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................X...X..X......XX.......................] duration: 5.772623906s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:56:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.772754345s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 49.53807ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 01:56:53 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 49.562566ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...X.................................] duration: 5.889033273s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:56:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.889070783s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 55.412846ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 01:56:17 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 55.437624ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.....................X...X..X......XX......................] duration: 5.892780167s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:55:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.892809053s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 46.50224ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 01:55:26 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 46.534951ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...X................................] duration: 5.647665732s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:55:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.647684126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X...X...............................] duration: 5.613120454s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:54:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.613172453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.....................X...X..X......XX.....................] duration: 6.00457221s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:54:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.004602658s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 71.130886ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 01:54:35 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 71.183075ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...X..............................] duration: 5.660845309s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:54:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.660942615s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.....................X...X..X......XX....................] duration: 5.629938409s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:54:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.629964409s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 361.649472ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 01:53:29 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 361.677857ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...X.............................] duration: 5.675102215s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:53:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.675118376s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.....................X...X..X......XX...................] duration: 6.001447712s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:53:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.001548785s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 220.641055ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 01:52:53 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 220.673736ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X.....................X...X..X......XX..................] duration: 6.296322994s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:52:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.296402855s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...X............................] duration: 5.648822745s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:52:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.648850028s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X...X...........................] duration: 5.677946075s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:51:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.67797494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X....X.....................X...X..X......XX.................] duration: 5.81821255s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:51:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.818250423s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...X..........................] duration: 5.746936653s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:51:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.746969465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X....X.....................X...X..X......XX................] duration: 7.904253829s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:50:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 7.904299857s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X....X.....................X...X..X......XX...............] duration: 12.968724014s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:50:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 12.968756485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...X.........................] duration: 7.479364638s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:50:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.479399406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...X........................] duration: 5.853160545s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:49:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.85319471s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 313.151583ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3918ada48f420b62 FAIL (2025-11-10 01:49:27 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33468->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 313.186409ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X....X.....................X...X..X......XX..............] duration: 10.434412562s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:49:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 10.434443101s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X...X.......................] duration: 7.122839741s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:48:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.122867985s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 304.307049ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3918ada48f420b62 FAIL (2025-11-10 01:48:42 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33468->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 304.334983ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 318.094254ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3918ada48f420b62 FAIL (2025-11-10 01:48:22 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33468->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 318.126966ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X....X.....................X...X..X......XX.............] duration: 6.124162135s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:48:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.124191452s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X...X......................] duration: 5.68155887s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:48:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.681586543s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 306.84803ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3918ada48f420b62 FAIL (2025-11-10 01:47:42 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33468->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 306.881342ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X....X.....................X...X..X......XX............] duration: 6.084232303s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:47:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.084281426s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 346.433037ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3918ada48f420b62 FAIL (2025-11-10 01:47:17 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33468->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 346.460198ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...X.....................] duration: 5.848369295s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:46:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.848398731s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 339.028017ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3918ada48f420b62 FAIL (2025-11-10 01:46:38 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33468->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 339.056151ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X....X.....................X...X..X......XX...........] duration: 5.951315147s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:46:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.951340305s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...X....................] duration: 5.624514763s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:46:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.624548827s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X....X.....................X...X..X......XX..........] duration: 5.756502045s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:45:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.756527466s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 697.06572ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3918ada48f420b62 FAIL (2025-11-10 01:45:28 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33468->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 697.090317ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...X...................] duration: 5.784188039s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:45:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.784216723s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 624.643482ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3918ada48f420b62 FAIL (2025-11-10 01:45:06 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:33468->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 624.673701ms core website

Orbiter Check History

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)