Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X...X..X......XX.........................................] duration: 5.726144911s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 02:09:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.72617096s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...X.....................................................] duration: 5.532730826s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 02:09:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.532762235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...X....................................................] duration: 5.830749475s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 02:09:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.830778902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X...X..X......XX........................................] duration: 5.854221452s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 02:08:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.854266207s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...X...................................................] duration: 5.705918229s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 02:08:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.705947475s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...X..................................................] duration: 5.803848765s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 02:07:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.803878061s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X...X..X......XX.......................................] duration: 5.695209734s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 02:07:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.695240844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X...X..X......XX......................................] duration: 11.887747765s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 02:07:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 11.887775297s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...X.................................................] duration: 7.318953088s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 02:06:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.319004636s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X...X..X......XX.....................................] duration: 9.778591438s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 02:06:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 9.778735353s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...X................................................] duration: 7.550711147s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 02:06:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.550737046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X...X..X......XX....................................] duration: 9.613935202s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 02:05:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 9.614113582s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X...X..X......XX...................................] duration: 6.002569163s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 02:05:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.002605964s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X...X...............................................] duration: 5.63379618s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 02:05:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.633823501s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X...X..............................................] duration: 5.602520668s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 02:04:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.602548779s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X...X..X......XX..................................] duration: 5.754689175s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 02:04:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.754784537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X...X.............................................] duration: 5.735425053s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 02:04:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.735455161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X...X..X......XX.................................] duration: 5.866051972s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 02:03:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.86608244s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 91.719722ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 02:03:51 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 91.737656ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X...X............................................] duration: 5.732475117s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 02:03:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.732501517s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 69.336739ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 02:03:16 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 69.36808ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X...X..X......XX................................] duration: 5.819921901s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 02:02:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.819959593s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 68.33884ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 02:02:29 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 68.368898ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X...X..X......XX...............................] duration: 5.986909447s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 02:02:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.98693805s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X...X...........................................] duration: 5.699764429s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 02:02:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.699792221s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X...X..........................................] duration: 5.668911539s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 02:01:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.668940414s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X...X..X......XX..............................] duration: 5.919404805s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 02:01:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.919436836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...X.........................................] duration: 5.700049651s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 02:01:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.70008062s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 50.763314ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 02:01:19 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 50.7987ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X...X..X......XX.............................] duration: 5.943035855s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 02:01:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.943066795s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 64.651899ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 02:00:36 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 64.679502ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X...X..X......XX............................] duration: 6.300392084s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 02:00:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.300414366s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...X........................................] duration: 5.840284754s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 02:00:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.840321885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X...X.......................................] duration: 5.738220313s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:59:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.738248525s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 60.180855ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 01:59:51 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 60.208458ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................X...X..X......XX...........................] duration: 5.73862695s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:59:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.738649432s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X...X......................................] duration: 5.687628892s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:59:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.687662756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................X...X..X......XX..........................] duration: 5.86784225s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:59:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.867869662s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X...X.....................................] duration: 5.735889639s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:59:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.735923252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................X...X..X......XX.........................] duration: 5.805172556s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:58:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.805202273s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 73.233461ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 01:58:37 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 73.259158ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X...X....................................] duration: 5.690502663s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:58:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.690531148s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 69.897357ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 01:58:12 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 69.924539ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X...X..X......XX........................] duration: 5.900839202s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:57:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.900869479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X...X...................................] duration: 5.641657461s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:57:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.641686306s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...X..................................] duration: 5.654536455s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:57:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.654570199s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 64.839397ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 01:57:11 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 64.865696ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................X...X..X......XX.......................] duration: 5.772623906s trace: https://jaeger.dev.storj.io/trace/2a20d00aaa1f7510 FAIL (2025-11-10 01:56:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 488]: Uploading piece... Nov 10 01:37:45.553891518 (17h33m44.105361979s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554161342 (17h33m44.105631803s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554478636 (17h33m44.105949098s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.554847398 (17h33m44.106317859s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555208326 (17h33m44.106678787s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.555572519 (17h33m44.10704298s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.556123017 (17h33m44.107593478s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557058970 (17h33m44.108529431s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.557616972 (17h33m44.109087433s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.559272345 (17h33m44.110742806s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:45.560871410 (17h33m44.112341881s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Uploading piece... Nov 10 01:37:47.331015302 (17h33m45.882485753s) [upload=15488]: Next returned nil segment Nov 10 01:37:47.331040148 (17h33m45.882510609s) [upload=15488]: Waiting for error group managing segments... Nov 10 01:37:47.610495877 (17h33m46.161966338s) [noise=true , address=77.90.31.19:4843 , piece_id=SLFM2NXS3GB343KFZA2P46SOB5CM4D37VB4OJ6YVVLS2Q7OK4OQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.655301566 (17h33m46.206772027s) [noise=true , address=77.90.31.10:4519 , piece_id=7W7TAXNHTYADJZMG3GDJUVS4W7F7B3RKSTMLOZZS2ZUJNP4A5WUA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:37:47.723708766 (17h33m46.275179238s) [noise=true , address=77.90.31.18:4806 , piece_id=DS3IBMXMCAM7FEQ2PAU2M3BPMV7ABUXXJF3NFMFJP2TI4JITT76A, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:10.440206114 (17h34m8.991676575s) [noise=true , address=145.79.227.14:4840 , piece_id=PGHKCOC5DF2RXHCVC7IIHFHYFDVU4L3KTYBKAOAVF3THVQRR56TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:11.549673278 (17h34m10.101143739s) [noise=true , address=145.79.227.10:4402 , piece_id=5GXGRJ5QCKRH2OI422YNAZ6O54DUJAXVQY5RZ34H6WL4TQ2QIVPQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.377791481 (17h34m12.929261942s) [noise=true , address=145.79.227.17:4971 , piece_id=2YEFK5S3Q24BWSR2ERNZHVNXWPFWBBS5PBLWGTTYYLSRRR7DNQRA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.548502610 (17h34m13.099973071s) [noise=true , address=145.79.227.22:4509 , piece_id=PEKXLP3437ZP2B2JMLR7EKT6LUPQDI4MGGTNOO3XGHK3QNDHWTQQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: <nil> Nov 10 01:38:14.601842358 (17h34m13.15331283s) [noise=true , address=145.79.227.10:4401 , piece_id=6LKWEHLJC3NTSDVHVAM3PHXVGGWX2ZBVP5BQLBQPHQZRJEWFM2CA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.10:4401: use of closed network connection Nov 10 01:38:14.602005329 (17h34m13.15347579s) [noise=true , address=145.79.227.10:4403 , piece_id=IVXADZL3ILFVIVREE6WXVDIA3T2ASPPVIH5GT4XXVFIT7LQD62TA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->145.79.227.10:4403: use of closed network connection Nov 10 01:38:14.602144925 (17h34m13.153615386s) [noise=true , address=145.79.227.31:4798 , piece_id=U3MNRGIUFB3H6TIHSIHSIJX5RAXDYUBA3O5LAP5UR2RKT67TZ37Q, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42826->145.79.227.31:4798: use of closed network connection Nov 10 01:38:14.602261466 (17h34m13.153731928s) [noise=true , address=145.79.227.30:4761 , piece_id=U7CW4DD7S4NVVQAM4BK3K6H6B3CTZS33BTW4Q2MK3K6NDDT4IZNA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40350->145.79.227.30:4761: use of closed network connection Nov 10 01:38:14.602362038 (17h34m13.153832509s) [noise=true , address=145.79.227.6:4258 , piece_id=IHGYNY24GDLUN7EBGX3Q6ZM27KKNNHRFV5UEJSWSW2DLUADZIMAQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50958->145.79.227.6:4258: use of closed network connection Nov 10 01:38:14.602598719 (17h34m13.1540692s) [noise=true , address=145.79.227.11:4732 , piece_id=4SHRFWORBCT66TXNPSAF4PYIZBAYWRJEEM7PTQRSVCEX6USG4VEQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52370->145.79.227.11:4732: use of closed network connection Nov 10 01:38:14.602710492 (17h34m13.154180953s) [noise=true , address=145.79.227.9:4366 , piece_id=MXENLBSTGC4N2LIEY55JC76L7GFICHZ5MS6OGTJYOO2ESXBTWTAA, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51824->145.79.227.9:4366: use of closed network connection Nov 10 01:38:14.602813959 (17h34m13.15428442s) [noise=true , address=145.79.227.13:4803 , piece_id=JQ7RM2SF7RC76CTQ6BA4E5OW6VHSWQ6LGCPBE46RTBPVWFYWIGXQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35400->145.79.227.13:4803: use of closed network connection Nov 10 01:38:14.603024369 (17h34m13.15449484s) [noise=true , address=145.79.227.28:4726 , piece_id=GSQAYOXB6VE7JUTQTM4D7COBMSZHIFBH7TXI3LRRLHIBDJFBRXOQ, seg_pos={0 0}, upload=15488]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60260->145.79.227.28:4726: use of closed network connection Nov 10 01:38:14.603054657 (17h34m13.154525118s) [seg_pos={0 0}, upload=15488]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 10 01:38:14.847241378 (17h34m13.398711839s) [upload=15488]: Done uploading segments... Nov 10 01:38:14.847279831 (17h34m13.398750282s) [upload=15488]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.772754345s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 49.53807ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/72e842be522c01c2 FAIL (2025-11-10 01:56:53 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 49.562566ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...X.................................] duration: 5.889033273s trace: https://jaeger.dev.storj.io/trace/2e3fa1015ac5f919 FAIL (2025-11-10 01:56:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386050603 (17h25m45.937521064s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386230996 (17h25m45.937701457s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386634986 (17h25m45.938105447s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.386663360 (17h25m45.938133821s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387066848 (17h25m45.938537309s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387180665 (17h25m45.938651126s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387470297 (17h25m45.938940758s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387763925 (17h25m45.939234386s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.387861361 (17h25m45.939331822s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388399095 (17h25m45.939869567s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388444351 (17h25m45.939914812s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388820949 (17h25m45.9402914s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.388947359 (17h25m45.94041782s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389231139 (17h25m45.94070159s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.389514750 (17h25m45.940985211s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:47.390090596 (17h25m45.941561057s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Uploading piece... Nov 10 01:29:48.975731178 (17h25m47.527201639s) [upload=15375]: Next returned nil segment Nov 10 01:29:48.975747068 (17h25m47.527217529s) [upload=15375]: Waiting for error group managing segments... Nov 10 01:29:49.384408290 (17h25m47.935878751s) [noise=true , address=45.91.116.9:4270 , piece_id=XPHSQXV3WD4G3A7PLOJMYXVWM65OBB4E5SQ7KABSIXUIPIA7VYWQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.403576249 (17h25m47.9550467s) [noise=true , address=77.90.31.14:4695 , piece_id=OHNPL2I7FUQPCFS6KUJIDXLK2X5IVKDU4TRSOQP6ZCHRKVOUUBAQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.412393298 (17h25m47.963863759s) [noise=true , address=77.90.31.11:4569 , piece_id=HUEWQPNZUQDV5SHPQ2BLDZU7SER6U43E6NL3O5LJY5RV2LKMGORA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414548141 (17h25m47.966018612s) [noise=true , address=45.91.116.8:4064 , piece_id=YNJMZGVQDAPLPET5TTI4KAUSNCUPBD6SDC4OIO2CNGBLFLDZQMBQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.414842132 (17h25m47.966312593s) [noise=true , address=77.90.31.19:4850 , piece_id=O2FLYGYV5VGHLPGRX4MP2FTPCFXM6I773UY2MXCW66RSELPDFD6Q, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.423757989 (17h25m47.97522845s) [noise=true , address=45.91.116.10:4109 , piece_id=4GBHD5P4XRMSCZA36CJUZ3VS6NPDW5BPBWOFXSMNM6J2OTZKVNJA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.425035943 (17h25m47.976506404s) [noise=true , address=77.90.31.12:4619 , piece_id=SI2FWNPIM7DQMCPJ2AUDMO4FEEBAV2EI5L2VHGSNYWF34H6TUPWA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.455553634 (17h25m48.007024095s) [noise=true , address=77.90.31.15:4730 , piece_id=ROEIOEXO6V7B5XQHOJWYR6UTPOB5CPUNZIW4DOBXZQKQWBL5CTPA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.555590334 (17h25m48.107060795s) [noise=true , address=5.231.73.3:4006 , piece_id=GROGN3APAMODJ74OFXEET24ORPMNKJUU7QKFS4YOQ27Q4JIIPLCA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:49.823765212 (17h25m48.375235663s) [noise=true , address=5.231.73.5:4023 , piece_id=IA7IDREFYXIUQWBWXE47IK7FVEBTVPOPJLZAG62FY45SJYHV7SLA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:29:50.075474846 (17h25m48.626945297s) [noise=true , address=5.231.73.3:4026 , piece_id=XYSHZN7ZBLRRUK7X7E3R5LXVP6UUPTIF2RWVWP7T4EJOJRTPAXOA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:39.676672149 (17h26m38.22814261s) [noise=true , address=145.79.227.27:4699 , piece_id=NVFKZWWR6UHG22FQFC67KXUTCDITDXNFBZ4LEFHMOIFB6EWQWRVA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: <nil> Nov 10 01:30:46.399643865 (17h26m44.951114326s) [noise=true , address=145.79.227.49:4346 , piece_id=NNQ4SJSQ3NQV42MBMLCVOKYLFQL6L22K7JWF5M6MQNWYWI7FVXSA, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:145.79.227.49:4346): protocol: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59888->145.79.227.49:4346: use of closed network connection Nov 10 01:30:46.399680044 (17h26m44.951150495s) [noise=true , address=145.79.227.16:4940 , piece_id=SVYMTN4VXP6V5LIXD6LBKV5KMALH44TYHJV3LG6F6JSMGR2IFNIQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12mPAwJqTNf7V3U291WVZNzMf3Tzi1fb4cQiJUEB8RH99xN13n, address:145.79.227.16:4940): protocol: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40574->145.79.227.16:4940: use of closed network connection Nov 10 01:30:46.399834598 (17h26m44.951305059s) [noise=true , address=145.79.227.31:4818 , piece_id=JLRRCWYDTLZYUIM7EBOYQHTDLXSQKKQ4BAZ7B72JC66G7KZQS7VQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:145.79.227.31:4818): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 01:30:46.399843274 (17h26m44.951313725s) [noise=true , address=145.79.227.7:4322 , piece_id=QXH6JLJJDOPWWJROXYJZPQX5RRXCPIRZDD7L4VPXB3UYYQCDHXKQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:145.79.227.7:4322): protocol: expected piece hash; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:46106->145.79.227.7:4322: use of closed network connection; EOF Nov 10 01:30:46.400020922 (17h26m44.951491393s) [noise=true , address=145.79.227.52:4479 , piece_id=ZCG7YLX3YHGEN2XPF6VJO4UFL5LEDAXKO6ITZLMGSNNPSVLHJZ4A, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:14Nnr21538kCTXCDrzPpSHYWAsnd85aT3A3jKPyLcuMvSShjAW, address:145.79.227.52:4479): protocol: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33170->145.79.227.52:4479: use of closed network connection Nov 10 01:30:46.400059325 (17h26m44.951529776s) [noise=true , address=145.79.227.18:4039 , piece_id=MB5UQJKFDAXLGY5WYAPLP46D2UW5VQP44Q3AWJNBJWYYRG3QIONQ, seg_pos={0 0}, upload=15375]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38412->145.79.227.18:4039: use of closed network connection Nov 10 01:30:46.400107377 (17h26m44.951577828s) [seg_pos={0 0}, upload=15375]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 10 01:30:46.655324672 (17h26m45.206795143s) [upload=15375]: Done uploading segments... Nov 10 01:30:46.655369007 (17h26m45.206839468s) [upload=15375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.889070783s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)