Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.460021236s FAIL (2025-10-06 22:12:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.460052415s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.429800782s FAIL (2025-10-06 22:12:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.429828565s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.982492471s FAIL (2025-10-06 22:12:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.982522917s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.105257845s FAIL (2025-10-06 22:12:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.105299654s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.742645411s FAIL (2025-10-06 22:12:00 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.742745973s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.834964559s FAIL (2025-10-06 22:11:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.83501772s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.272119099s FAIL (2025-10-06 22:11:07 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.272156059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.607309882s FAIL (2025-10-06 22:11:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.607345319s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.415860518s FAIL (2025-10-06 22:11:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.415968835s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.410955929s FAIL (2025-10-06 22:10:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.410981828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.247418011s FAIL (2025-10-06 22:10:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.247444461s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.512790933s FAIL (2025-10-06 22:10:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.512826079s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.440175879s FAIL (2025-10-06 22:10:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.440196578s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 692.481582ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login FAIL (2025-10-06 22:09:55 UTC) orbiter warning: Get "https://us1.storj.io/login": context deadline exceeded Duration: 692.513071ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.386317256s FAIL (2025-10-06 22:09:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.386339479s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.483843572s FAIL (2025-10-06 22:09:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.483867337s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.157528858s FAIL (2025-10-06 22:09:36 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.157559406s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 685.367686ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login FAIL (2025-10-06 22:09:35 UTC) orbiter warning: Get "https://us1.storj.io/login": context deadline exceeded Duration: 685.388414ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 689.713983ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login FAIL (2025-10-06 22:09:16 UTC) orbiter warning: Get "https://us1.storj.io/login": context deadline exceeded Duration: 689.747526ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 693.040257ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login FAIL (2025-10-06 22:08:52 UTC) orbiter warning: Get "https://us1.storj.io/login": context deadline exceeded Duration: 693.061967ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.674518727s FAIL (2025-10-06 22:08:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.674554334s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.433161041s FAIL (2025-10-06 22:08:37 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.433191639s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.793432512s FAIL (2025-10-06 22:08:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.793469973s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.74536232s FAIL (2025-10-06 22:08:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.74539429s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 694.893092ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login FAIL (2025-10-06 22:08:13 UTC) orbiter warning: Get "https://us1.storj.io/login": context deadline exceeded Duration: 694.91214ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.668338984s FAIL (2025-10-06 22:08:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.668366235s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.478248539s FAIL (2025-10-06 22:07:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.47828076s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.279302865s FAIL (2025-10-06 22:07:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.279327693s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 718.164125ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login FAIL (2025-10-06 22:07:32 UTC) orbiter warning: Get "https://us1.storj.io/login": context deadline exceeded Duration: 718.197149ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.492810055s FAIL (2025-10-06 22:07:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.492841855s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.205337103s FAIL (2025-10-06 22:07:20 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.205370104s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.471117127s FAIL (2025-10-06 22:07:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.471146743s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.418289864s FAIL (2025-10-06 22:07:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.418326804s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.54367195s FAIL (2025-10-06 22:06:54 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.543704864s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.825423148s FAIL (2025-10-06 22:06:53 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.825453686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.999571941s FAIL (2025-10-06 22:06:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.999884767s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.197532832s FAIL (2025-10-06 22:06:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.197558663s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 694.015443ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login FAIL (2025-10-06 22:06:27 UTC) orbiter warning: Get "https://us1.storj.io/login": context deadline exceeded Duration: 694.041654ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.692445731s FAIL (2025-10-06 22:06:17 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.692480156s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 7.02903827s FAIL (2025-10-06 22:06:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.029066644s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.440699335s FAIL (2025-10-06 22:05:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.440735775s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 7.105443491s FAIL (2025-10-06 22:05:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 7.105483146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.221098096s FAIL (2025-10-06 22:05:27 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.221130908s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 700.973782ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login FAIL (2025-10-06 22:05:27 UTC) orbiter warning: Get "https://us1.storj.io/login": context deadline exceeded Duration: 701.019389ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 39.583220744s FAIL (2025-10-06 22:05:20 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 39.583252385s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.240728234s FAIL (2025-10-06 22:05:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.240790253s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 12.146917435s FAIL (2025-10-06 22:05:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 12.146950748s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 726.556232ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login FAIL (2025-10-06 22:04:54 UTC) orbiter warning: Get "https://us1.storj.io/login": context deadline exceeded Duration: 726.579958ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.058580649s FAIL (2025-10-06 22:04:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.058605717s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.24170171s FAIL (2025-10-06 22:04:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.24173262s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)