Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.75717014s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:49:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.757200628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.081409812s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:49:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.081443759s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.572991497s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:48:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.573023607s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.932182679s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:48:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.932194571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.985566094s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:48:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.985598795s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.796558593s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:47:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.796612915s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.130968589s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:47:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.131002283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.631392594s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:47:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.631421219s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.963718213s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:47:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.963738021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.894586186s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:46:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.894610522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.919939837s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:46:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.919982839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.848998092s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:45:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.84902923s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.103080755s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:45:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.103111673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.836415543s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:45:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.836446201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.981870199s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:44:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.981911458s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.853989223s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:44:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.854020651s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.011615349s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:44:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.011646829s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.893013457s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:43:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.893043004s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.847899919s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:43:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.847947298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.991948731s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:42:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.991978316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.822864701s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:42:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.822892593s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.941346543s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:42:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.941383144s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.797654651s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:42:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.797682163s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.959412512s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:41:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.959523803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.828526068s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:41:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.82856941s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.860362901s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:40:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.860387046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.987995608s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:40:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.988032449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.90035195s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:40:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.900379252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.793777334s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:40:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.793796761s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...........................................X...............] duration: 6.026358053s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:39:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.026386418s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.859532558s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:39:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.859582272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...........................................X..............] duration: 6.038243792s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:39:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.038277336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.792432887s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:38:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.792461943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...........................................X.............] duration: 5.971008302s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:38:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.971038209s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...........................................X............] duration: 5.902154698s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:37:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.902180858s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.888606684s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:37:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.888669964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...........................................X...........] duration: 6.126320087s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:36:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.126346307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.902516968s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:36:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.902549751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...........................................X..........] duration: 6.10419974s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:35:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.104220118s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.790339028s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:34:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.790367272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...........................................X.........] duration: 5.9209972s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:34:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.921032948s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.725474183s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:34:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.725505393s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...........................................X........] duration: 5.98046081s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:33:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.980487612s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X...........................................X.......] duration: 6.076801669s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:33:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.076837508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.766873573s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:33:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.7669035s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.893075101s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:32:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.893108725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X...........................................X......] duration: 6.074016149s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:32:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.074045104s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X...........................................X.....] duration: 6.097116487s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:32:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.097144952s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.725147072s trace: https://jaeger.dev.storj.io/trace/7aac9668bee4a964 FAIL (2025-11-14 04:32:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=102391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 14 04:27:57.469343915 (116h23m56.020814366s) [upload=102391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 14 04:27:57.727534683 (116h23m56.279005144s) [seg_pos={0 0}, upload=102391]: Begin upload segment... Nov 14 04:27:57.727605278 (116h23m56.279075739s) [seg_pos={0 0}, upload=102391]: Done begin upload segment. Nov 14 04:27:57.727641026 (116h23m56.279111487s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.728032522 (116h23m56.279502983s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.729613532 (116h23m56.281084083s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.731458655 (116h23m56.282929116s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.732537070 (116h23m56.284007531s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.733574995 (116h23m56.285045456s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735130728 (116h23m56.286601199s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.735616894 (116h23m56.287087345s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.737740838 (116h23m56.289211299s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.738385065 (116h23m56.289855516s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.739648160 (116h23m56.291118621s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.740446620 (116h23m56.291917082s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.741126735 (116h23m56.292597196s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.742683470 (116h23m56.294153921s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.744210047 (116h23m56.295680508s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.745511845 (116h23m56.296982306s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.746104333 (116h23m56.297574794s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:57.748079675 (116h23m56.299550136s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Uploading piece... Nov 14 04:27:59.104919416 (116h23m57.656389877s) [upload=102391]: Next returned nil segment Nov 14 04:27:59.104938111 (116h23m57.656408582s) [upload=102391]: Waiting for error group managing segments... Nov 14 04:27:59.465536154 (116h23m58.017006615s) [noise=true , address=64.49.13.155:4924 , piece_id=WESOY7WYSY2T3FXRAT7JFRIHL4A3USRA4FLGOAWN52FUWBDKB2BQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.469442893 (116h23m58.020913354s) [noise=true , address=5.231.73.5:4021 , piece_id=LH43FLZLZ2HO7JCMQH6NFJBWWXSZFVKHNP745Q5YR4BX7XFV6UYQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.472429661 (116h23m58.023900122s) [noise=true , address=5.231.73.4:4003 , piece_id=S25ILIXL45BQTREUHKKSHBVRHGMLKVB7PPX353VQLIYPS3SM47HA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.476606474 (116h23m58.028076935s) [noise=true , address=64.49.13.26:4952 , piece_id=ZURUTSDRWJTBVRYD5KKUXOHDJXXWMKAPYWN7UPVBQP2APEXTDLQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.478936251 (116h23m58.030406712s) [noise=true , address=64.49.13.140:4505 , piece_id=NZC6UUJQE4JSJKRX3H376V3BE55P36L64H4BRHWIHZCMVCOU3EUQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.542344724 (116h23m58.093815185s) [noise=true , address=5.231.73.2:4016 , piece_id=XNRIIAHH7C3YWM7YWVSUMMN7LYFG3XYSM76BPNEOGRD2GO6OQE5Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.677332408 (116h23m58.228802869s) [noise=true , address=77.90.31.18:4820 , piece_id=WSLQMSKH7QMOESQAIDXGZUJ2V7BLBE3KFAY2QKZU3PGT2KAC6WGA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.729640212 (116h23m58.281110673s) [noise=true , address=45.91.116.4:4954 , piece_id=Z7XBF3IP7JJCNHTSKC4YHYTDSAR5ZMTPHFTVLPKCFPEDTXHAMUBA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.732364741 (116h23m58.283835192s) [noise=true , address=45.91.116.14:4208 , piece_id=HR4YSVK6KCH437P7RC5VSM45AOEIMX3UTU4NPI4VRD4LBFM4YLUA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.748117227 (116h23m58.299587688s) [noise=true , address=77.90.31.13:4651 , piece_id=HCNKA33DJV4NECBAXJMRNSKK4S54MACLQJBOBLTNSSJBWBP7IMMQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756385441 (116h23m58.307855903s) [noise=true , address=77.90.31.19:4867 , piece_id=242BN4DA2CMNEGRKHXVU6ZTCKMGOOWK6OQ3VZS6WEQPZ2OMDZ57Q, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.756736440 (116h23m58.308206901s) [noise=true , address=77.90.31.14:4687 , piece_id=YKTLDLCW46WZ4JE3KOB66R6ZMKIE5L7ZEOGBYM5ISS7Y35TYGJHA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:27:59.763406992 (116h23m58.314877443s) [noise=true , address=45.91.116.7:4027 , piece_id=WTNKB3CSZ5MTVNW2QQGX7USPK5VFAJ2ND7MXRQIZYJBINMK57CXQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: <nil> Nov 14 04:28:56.756904418 (116h24m55.308374889s) [noise=true , address=145.79.227.37:4571 , piece_id=3FMYRCBADPZ5H2HVY3ZVYLREXEA6X34YVFYUCKRRR6I55KLBIDSA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1kL49w1XE8S2Dgk9RUfjuXFxisMW3BUEPxPZR87TCVBpAwbyit, address:145.79.227.37:4571): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.757377489 (116h24m55.30884798s) [noise=true , address=5.231.73.3:4036 , piece_id=2ITT5NKPLI6JNSIOF63IJUIK2S4JIV37LTSMYOAIOPXWKE4B47DQ, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1Lthpe75uKC8ob2p2HSLiYHE9fYWvYayu3cnaQ7PnduXCDZFDv, address:5.231.73.3:4036): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.758144910 (116h24m55.309615411s) [noise=true , address=145.79.227.33:4879 , piece_id=YNQHCBAS4R2JM2XIEKGBVCIWFP2SWLP254EDZZJMV6IDAFXOPQQA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:145.79.227.33:4879): protocol: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41332->145.79.227.33:4879: use of closed network connection Nov 14 04:28:56.758502852 (116h24m55.309973313s) [noise=true , address=145.79.227.49:4356 , piece_id=BCWGJ5RCV3Q3WY2NEULC6VBWWINWFBZJ5RYJ2RW6MMDL4RD3LACA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1, address:145.79.227.49:4356): protocol: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58568->145.79.227.49:4356: use of closed network connection Nov 14 04:28:56.760031634 (116h24m55.311502095s) [noise=true , address=145.79.227.38:4984 , piece_id=WTZA4OIOZ2COFFMD6MOQY4HTNQNOVYA3IPI37AGVEX577KEI3QLA, seg_pos={0 0}, upload=102391]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:145.79.227.38:4984): protocol: expected piece hash; context deadline exceeded; EOF Nov 14 04:28:56.760147805 (116h24m55.311618266s) [seg_pos={0 0}, upload=102391]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 04:28:57.013775143 (116h24m55.565245604s) [upload=102391]: Done uploading segments... Nov 14 04:28:57.013846178 (116h24m55.565316639s) [upload=102391]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.725173303s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X...........................................X....] duration: 6.006490938s trace: https://jaeger.dev.storj.io/trace/3fc1c53eb1964077 FAIL (2025-11-14 04:31:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757037132 (116h23m56.308507583s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.757551642 (116h23m56.309022113s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758202921 (116h23m56.309673384s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.758643932 (116h23m56.310114393s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759202154 (116h23m56.310672616s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759256859 (116h23m56.31072732s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.759600383 (116h23m56.311070844s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760030752 (116h23m56.311501214s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.760602551 (116h23m56.312073022s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.761033622 (116h23m56.312504083s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762283092 (116h23m56.313753553s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.762901770 (116h23m56.314372231s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:57.764606997 (116h23m56.316077458s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Uploading piece... Nov 14 04:27:59.569346501 (116h23m58.120816962s) [upload=102392]: Next returned nil segment Nov 14 04:27:59.569367190 (116h23m58.120837651s) [upload=102392]: Waiting for error group managing segments... Nov 14 04:27:59.858609156 (116h23m58.410079617s) [noise=true , address=77.90.31.13:4627 , piece_id=C7TTGKTJKSHOXRLZSHJOI6KSP3D53QKKVBJK45227VUZWGPYI2HA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.885377208 (116h23m58.436847669s) [noise=true , address=45.91.116.5:4298 , piece_id=GT3APYUXLNMNKUTTKODR7FJOPDQFGSRHJLLXQNZSHSAP6HULBPRA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.891248067 (116h23m58.442718528s) [noise=true , address=45.91.116.14:4189 , piece_id=D6B4PX66G52TVGCPVKEZAM4TASMXZQN4A2T4OIGTKF4EKZKCYQNQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.931472914 (116h23m58.482943375s) [noise=true , address=77.90.31.12:4589 , piece_id=S6ME2FF5OOJIUE7NC6IIHVMGUGYWOHSIFBB4MEA6AYKJPJQYGKFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.955837770 (116h23m58.507308231s) [noise=true , address=45.91.116.8:4047 , piece_id=RQUOFV73RH6IQQ3PFAPKXVUV2XCL4SPZETNCV7ZINT2KOOM7KH6A, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:27:59.957935033 (116h23m58.509405494s) [noise=true , address=45.91.116.13:4155 , piece_id=MWN4U456OSZSEZP4BIYE34MWC26CNGO2B2KC6JXTVN2JCDE56DZQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: <nil> Nov 14 04:28:26.800302861 (116h24m25.351773332s) [noise=true , address=145.79.227.31:4797 , piece_id=5U5WDETNYGZEDAQVX3ZT4GYHUK2TXNTHXKTIS2QRDZZDKDMQTKLQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50378->145.79.227.31:4797: use of closed network connection Nov 14 04:28:26.800403783 (116h24m25.351874244s) [noise=true , address=145.79.227.32:4834 , piece_id=EDXCHV2NLLH6A7O6OKAW2DIJPMNQQRTBCGWNJRDL65ZWBOYV3OPQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50760->145.79.227.32:4834: use of closed network connection Nov 14 04:28:26.800528972 (116h24m25.351999433s) [noise=true , address=145.79.227.11:4730 , piece_id=YOJ6ZY534GOA7HPNMKOG57VMPIIUGZ2KX2PWX3XDKHIVVD6SYRFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52672->145.79.227.11:4730: use of closed network connection Nov 14 04:28:26.800628021 (116h24m25.352098492s) [noise=true , address=145.79.227.17:4970 , piece_id=6DLEUTVGJGC72CDCAML3FUO63ROWU4247ESWPRAS3J46STYBCNEQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:35302->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.800702292 (116h24m25.352172753s) [noise=true , address=145.79.227.28:4725 , piece_id=3VJLR766RMRBMGOJ54IIEO7W3P4EQTVSUD4ILTUPS2JR6IJFV4QQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: read tcp 188.166.225.170:55540->145.79.227.28:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801179060 (116h24m25.352649511s) [noise=true , address=145.79.227.27:4691 , piece_id=2IPDVPH3EEHN4LI2NNNB3O75S2RKA3KRZ2XWSAXZVQNX53Z3Y77Q, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: read tcp 188.166.225.170:54006->145.79.227.27:4691: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801189189 (116h24m25.35265965s) [noise=true , address=145.79.227.23:4547 , piece_id=4YUR4LSPJTXG3KNZLRC5WTZOKQCYHH3BK5EQPOOZ6XWVG5LXJ2PA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: read tcp 188.166.225.170:45138->145.79.227.23:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801349285 (116h24m25.352819756s) [noise=true , address=145.79.227.20:4437 , piece_id=W3DFS6642EPG5YTWGJF4H2WKEX6GOZLHXM3BO2E2BV62SFC5MOVA, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: read tcp 188.166.225.170:43170->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801491255 (116h24m25.352961726s) [noise=true , address=145.79.227.4:4186 , piece_id=IKNVPA3U3V4W445MAXXCTPMAE5PRW5FJHU4DOCCPFT6M7HPCZDFQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: read tcp 188.166.225.170:45138->145.79.227.4:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801526011 (116h24m25.352996472s) [noise=true , address=145.79.227.6:4259 , piece_id=ZSFJH3GY6A3HLAHCZ57PTZOOKW53ZX3KK2RK2UA6RBVCD6GT3KXQ, seg_pos={0 0}, upload=102392]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:37190->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 04:28:26.801558142 (116h24m25.353028593s) [seg_pos={0 0}, upload=102392]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 14 04:28:27.045452737 (116h24m25.596923198s) [upload=102392]: Done uploading segments... Nov 14 04:28:27.045492633 (116h24m25.596963094s) [upload=102392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.00652323s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)