Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.711367903s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:40:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.711394713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.768568663s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:40:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.768595325s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.657127375s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:39:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.657143624s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.68482811s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:38:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.684853037s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.789208465s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:37:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.789236078s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.757641478s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:37:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.757674341s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.767603288s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:36:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.767619238s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.759964064s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:35:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.759990915s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.727069197s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:35:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.727095097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.72479589s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:34:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.724822611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.813570633s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:33:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.813594939s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.849376949s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:32:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.849404702s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.822356481s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:31:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.822379995s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.799856449s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:31:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.799880485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.914793386s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:31:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.914889849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.778102658s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:30:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.778130843s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.727302974s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:29:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.727330887s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.95619045s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:28:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.956211981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.722351552s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:27:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.722375839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.799030908s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:27:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.799062688s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.729468801s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:26:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.729492115s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.728398906s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:25:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.728422029s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.754915504s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:24:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.754939549s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.638067263s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:23:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.638093924s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.774245941s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:22:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.774344911s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.774282873s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:21:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.774336636s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.665541774s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:20:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.665573453s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 8.694407819s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:19:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.69443518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.75996465s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:18:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.759988544s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.62323661s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:17:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.62326258s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.743208754s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:17:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.743233561s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.243688467s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:17:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.243713825s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 4.738081194s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 22:02:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.738106772s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 4.735254951s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 22:01:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.735274188s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 4.339793666s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 22:00:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.339833302s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 4.627005872s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 21:59:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.627022282s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 4.680885484s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 21:58:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.680915702s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 4.65805322s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 21:58:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.658081383s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 4.688751635s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 21:57:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.688784599s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 4.195449022s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 21:56:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.195537861s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 4.250898272s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 21:55:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.250925434s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 4.703771092s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 21:54:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.703799708s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 4.707429639s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 21:53:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.707455387s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 4.741969886s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 21:53:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.742192399s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 4.670983702s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 21:53:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.671009491s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 4.296813308s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 21:52:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.296838917s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 4.710187613s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 21:51:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.710219262s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 4.636984545s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 21:50:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.637020865s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 4.70135795s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 21:49:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.701386544s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 4.678390624s trace: https://jaeger.dev.storj.io/trace/7558e744ce4160c7 FAIL (2025-12-25 21:48:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== piece... Dec 25 21:13:56.771132713 (485h44m56.642215748s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771290233 (485h44m56.642373268s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771424147 (485h44m56.642507182s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.771655458 (485h44m56.642738493s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772080678 (485h44m56.643163713s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772389887 (485h44m56.643472922s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.772690830 (485h44m56.643773865s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773055754 (485h44m56.644138789s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:56.773366186 (485h44m56.644449221s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Uploading piece... Dec 25 21:13:57.595106368 (485h44m57.466189413s) [upload=426336]: Next returned nil segment Dec 25 21:13:57.595119322 (485h44m57.466202357s) [upload=426336]: Waiting for error group managing segments... Dec 25 21:13:57.976476392 (485h44m57.847559437s) [noise=true , address=81.171.14.70:4028 , piece_id=N5PE7WM3LXALODMWG4PE5GJY4JTC4LE55EIYFG3E2ATIFIQ7REUQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.984058819 (485h44m57.855141864s) [noise=true , address=185.167.204.205:4022 , piece_id=XQTN7N662BPAJN2HAGJVNB7KV4HVJ3MDHI52K6VVQXYWJ6OQ2FLA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.985179434 (485h44m57.856262469s) [noise=true , address=185.167.204.206:4053 , piece_id=HKLIP347QTNQWBQOKV3JSFGNOSJXY65NY5NWHNOD4CULC6HND2TQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986020966 (485h44m57.857104001s) [noise=true , address=185.167.204.205:4001 , piece_id=XHI54JXIF553XP3O75YFCMULFRKGQKCZMODK4O7TQPZ6GNQ76U4A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.986357256 (485h44m57.857440291s) [noise=true , address=185.167.204.205:4026 , piece_id=L4OL4FBJAQIPJZOLMS7HQDLS5ANAB5BH4JBXWA2YOUIMWH3ZMGEA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.988328560 (485h44m57.859411595s) [noise=true , address=185.167.204.205:4029 , piece_id=GWICARJW2AWFF6N2L2BVOYTEWI6PBB2PWOI2MHRBBAJVTCXPMO6A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.989905162 (485h44m57.860988197s) [noise=true , address=185.167.204.205:4027 , piece_id=5CUZIUOOQ7CKYVAOMH24DRP5CZMUDUUBPAZYWIEDMCDAQJMDVM2Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.990003198 (485h44m57.861086233s) [noise=true , address=185.167.204.205:4036 , piece_id=NGZXM5UHPAVWMPQSJEQ4RNDBLZ4PMJDWB5XJHXQSEPL663TJ56AQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.992957895 (485h44m57.86404093s) [noise=true , address=81.171.14.70:4047 , piece_id=NXQAQS3QIK4CLW5B3UZYDMIZ2YBH6XF6Q3ZRZGEBQYC54GS42BWQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.993962587 (485h44m57.865045622s) [noise=true , address=185.167.204.206:4067 , piece_id=WDQCOP324WE7A6CEX23SF2HKIYBKCALSI62UU3HZ6NWXU54P3OJA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.994190371 (485h44m57.865273406s) [noise=true , address=185.167.204.206:4062 , piece_id=V4UX5YBPKJGIOI2FB7YXIH5ZBJSMTOCUBPEKFTZZVXKKZKFSVFMA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995280166 (485h44m57.866363201s) [noise=true , address=185.167.204.206:4047 , piece_id=3ON2YNYSFJTO3KSTRHLR62DZ25CHP7MNEOTYGXL64C2XJHSJEMCA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.995367914 (485h44m57.866450959s) [noise=true , address=185.167.204.206:4064 , piece_id=LFM3UKVFSNQYQIO3DJITC2EYVGBQD2JXYPK4DD2IRL2KGQG2BAUA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:57.997243275 (485h44m57.86832631s) [noise=true , address=185.167.204.206:4044 , piece_id=ZC6ZTJSS27D47GG6H72TIPCU5I6XYSPUAO6BEK7FO6GDGA3DAS7Q, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.009518258 (485h44m57.880601303s) [noise=true , address=185.167.204.205:4010 , piece_id=2HDMNDKVDC6H63R64AZLVITJGPJ6DYLWIGTSXABX5F7JCG3RJXHA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.011034034 (485h44m57.882117079s) [noise=true , address=185.167.204.205:4033 , piece_id=FPMOUO7ODH5HGX5TQMI3B4ZJNBFJDJKYEJNBYG6UL63PYAWTYZAA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014221173 (485h44m57.885304218s) [noise=true , address=185.167.204.205:4008 , piece_id=S535IAWODU464PEYL22IMLBEKKSCPH32ZYKLLI2BCDOJSHSPF3PA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.014480747 (485h44m57.885563782s) [noise=true , address=185.167.204.205:4030 , piece_id=2OFHY32EJQANORDZBVIWBOPIXJDJUO5JYLERNLJW24YC32YCSHJQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.017501679 (485h44m57.888584714s) [noise=true , address=185.167.204.206:4055 , piece_id=JYTVB64KJU6NTT2OKTBTYM3ITN6LN6JTZWXMZ5RHQRI4FIWQ67WA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.020812724 (485h44m57.891895759s) [noise=true , address=81.171.14.70:4046 , piece_id=U4WOOXI2LPVNBKQXEBCVAJELLYCJNJMQJ6MUIOX6JDKZ5U3IWCRQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.036333589 (485h44m57.907416624s) [noise=true , address=81.171.14.70:4044 , piece_id=RALKLQIQCJA4VLDTZT6FC6QH2YKQBVDSN3FJTYMIATXLDZKSELQQ, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:58.567781283 (485h44m58.438864318s) [noise=true , address=81.171.14.70:4032 , piece_id=2JXQ7J7PGIFKAKOQPMDPKFN52NGQ3YGF2KVAKAACYELI7T6USSVA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:13:59.859284313 (485h44m59.730367348s) [noise=true , address=89.149.201.147:4017 , piece_id=WGJTVV3POL35LZM72ZUVFNARESTWPM6UEBBIYAIBL4TCMTZAY5HA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:04.290883050 (485h45m4.161966105s) [noise=true , address=89.149.201.147:4014 , piece_id=35CRPLZ6JGGOQZFWSOJLXZERN7WDC4VXN2LSZXL64GUINZUHQ7EA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: <nil> Dec 25 21:14:56.128630221 (485h45m55.999713266s) [noise=true , address=89.149.201.147:4010 , piece_id=SFNFKVMQLQZFMTXJM4F6JK3G47P25NN3PJMZH4632636TPGBGR7A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9, address:89.149.201.147:4010): protocol: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60900->89.149.201.147:4010: use of closed network connection Dec 25 21:14:56.128772011 (485h45m55.999855046s) [noise=true , address=89.149.201.147:4007 , piece_id=3LASTPWFFY5LNKWBDEFGJC3ISDKXSFLD5WTF4WJDJUPVESHUMI5A, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54090->89.149.201.147:4007: use of closed network connection Dec 25 21:14:56.128938387 (485h45m56.000021422s) [noise=true , address=89.149.201.147:4020 , piece_id=Y3H3HY72RYGJXMFLQCNZ6QAS3S7B2YEH7NCQYBAXIRN345BBB5SA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: read tcp 188.166.225.170:50116->89.149.201.147:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129146574 (485h45m56.000229619s) [noise=true , address=89.149.201.147:4005 , piece_id=UKQM6IOQDT6ZGMKXP6NTWNJPR6DUCWELZ6YUQWECO72DJW3HWFYA, seg_pos={0 0}, upload=426336]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 188.166.225.170:53156->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 21:14:56.129178625 (485h45m56.00026166s) [seg_pos={0 0}, upload=426336]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Dec 25 21:14:56.294222363 (485h45m56.165305408s) [upload=426336]: Done uploading segments... Dec 25 21:14:56.294261267 (485h45m56.165344302s) [upload=426336]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.678416794s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)