Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.462244672s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:32:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.462270971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.575335562s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:32:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.57536498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.X] duration: 30.002450696s trace: https://jaeger.dev.storj.io/trace/78aad39bf14a8345 FAIL (2025-12-29 18:32:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: 4LW7HXKNSAPFKEXYNQF6B74Q6HJPALHI2XJ4EZPUY75XFUUED5UQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: XUQIEWBURLIUSQHIZY24OH3CYK4EDY5ZSWCJO6CM6FAEO44O7ZFA) context deadline exceeded ======LOGS====== 2.883644679s) [upload=507613]: Uploading segments... Dec 29 18:31:53.012656885 (579h2m52.88373994s) [upload=507613]: Got next segment. Inline: false Dec 29 18:31:53.012700308 (579h2m52.883783353s) [upload=507613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:31:53.012707301 (579h2m52.883790346s) [upload=507613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:31:53.259210981 (579h2m53.130294026s) [seg_pos={0 0}, upload=507613]: Begin upload segment... Dec 29 18:31:53.259274212 (579h2m53.130357247s) [seg_pos={0 0}, upload=507613]: Done begin upload segment. Dec 29 18:31:53.259312094 (579h2m53.130395129s) [noise=true , address=77.90.31.10:4520 , piece_id=UEI5HMGDDUXHH4WXFWUBZGDR5MBD4LGBNRMB24P4TLRYNMNPJIWA, seg_pos={0 0}, upload=507613]: Uploading piece... Dec 29 18:31:53.259363101 (579h2m53.130446166s) [noise=true , address=145.79.227.4:4185 , piece_id=ZLPC6AY3MSH2HNBOSNG5VJPSCLRZZT7AOIMZQE4VPPUXXAJOGBOA, seg_pos={0 0}, upload=507613]: Uploading piece... Dec 29 18:31:53.260036704 (579h2m53.131119739s) [noise=true , address=145.79.227.30:4761 , piece_id=SE7OVOLW6QYVEMIT2ON63BHZ7Q4HJI5P5XZVFL2O3WQP3AONRO6A, seg_pos={0 0}, upload=507613]: Uploading piece... Dec 29 18:31:53.260326816 (579h2m53.131409851s) [noise=true , address=145.79.227.46:4602 , piece_id=D77OTONDZY335R6K5RZMDEANFQP4CBE2NHXHF4KKYPDHQJ2AAF5A, seg_pos={0 0}, upload=507613]: Uploading piece... Dec 29 18:31:53.260703543 (579h2m53.131786589s) [noise=true , address=145.79.227.42:4121 , piece_id=XWUGAO5O25GICVOEXPBI2THLDLZCHYVSEAXQL4S6HBVTKULAXAPA, seg_pos={0 0}, upload=507613]: Uploading piece... Dec 29 18:31:53.261041217 (579h2m53.132124262s) [noise=true , address=145.79.227.37:4565 , piece_id=WFBSXJ2KPFM4W3J23G5ZCP2TQM3VL6TE76G2BDXXTVTKLG34HJWA, seg_pos={0 0}, upload=507613]: Uploading piece... Dec 29 18:31:53.261310569 (579h2m53.132393604s) [noise=true , address=145.79.227.26:4655 , piece_id=GTVOELMMLZ3CDKCPXYAO3IP3WYFYGEKW6TC5WV2BNJJWAJLKAETQ, seg_pos={0 0}, upload=507613]: Uploading piece... Dec 29 18:31:53.261556918 (579h2m53.132639953s) [noise=true , address=145.79.227.50:4375 , piece_id=5AHPGMCU7K2C63QQDG5ZAKILRHFNFT4RTWNHNOWP66PME4FFK7UA, seg_pos={0 0}, upload=507613]: Uploading piece... Dec 29 18:31:53.262016273 (579h2m53.133099308s) [noise=true , address=77.90.31.11:4555 , piece_id=4LW7HXKNSAPFKEXYNQF6B74Q6HJPALHI2XJ4EZPUY75XFUUED5UQ, seg_pos={0 0}, upload=507613]: Uploading piece... Dec 29 18:31:53.262359336 (579h2m53.133442371s) [noise=true , address=45.91.116.4:4939 , piece_id=EIGP3ZP53RHBVFVZPNUADEKX6S53NVO22D5SKNEQ2F6TMEXTFRJQ, seg_pos={0 0}, upload=507613]: Uploading piece... Dec 29 18:31:53.262492990 (579h2m53.133576036s) [noise=true , address=145.79.227.48:4303 , piece_id=7LSFLKWUFKD3A6LGR7THTWKTO7YSRUT7ZGRAPPSUZE3QXELSG6HQ, seg_pos={0 0}, upload=507613]: Uploading piece... Dec 29 18:31:53.262834030 (579h2m53.133917075s) [noise=true , address=145.79.227.38:4979 , piece_id=KYS4Q6FY5SVKBTGZCMMRGXKI6A5JBW6VT7MRCU3RVJQ3XHT4DGEQ, seg_pos={0 0}, upload=507613]: Uploading piece... Dec 29 18:31:53.262842997 (579h2m53.133926032s) [noise=true , address=145.79.227.12:4765 , piece_id=V366P2EAHA2PDAJNPF3DYMMS6DBXCKV53VSKH4NEUSE7VWOQV7CA, seg_pos={0 0}, upload=507613]: Uploading piece... Dec 29 18:31:53.263112160 (579h2m53.134195195s) [noise=true , address=45.91.116.12:4227 , piece_id=ZK22ZK4R32QW4KNNGDJ24HEXYUCDZYJ6TDPBQNEH7GAIGOGQKSNA, seg_pos={0 0}, upload=507613]: Uploading piece... Dec 29 18:31:53.263248079 (579h2m53.134331114s) [noise=true , address=145.79.227.45:4230 , piece_id=NB5F3WY5RG5ZVI4D4GXUR7VD3CYP63LRERUFS433M3NQDJZMRHHQ, seg_pos={0 0}, upload=507613]: Uploading piece... Dec 29 18:31:53.263492865 (579h2m53.13457591s) [noise=true , address=45.91.116.10:4082 , piece_id=XUQIEWBURLIUSQHIZY24OH3CYK4EDY5ZSWCJO6CM6FAEO44O7ZFA, seg_pos={0 0}, upload=507613]: Uploading piece... Dec 29 18:31:54.949115907 (579h2m54.820198952s) [upload=507613]: Next returned nil segment Dec 29 18:31:54.949128511 (579h2m54.820211546s) [upload=507613]: Waiting for error group managing segments... Dec 29 18:31:55.229201181 (579h2m55.100284226s) [noise=true , address=145.79.227.4:4185 , piece_id=ZLPC6AY3MSH2HNBOSNG5VJPSCLRZZT7AOIMZQE4VPPUXXAJOGBOA, seg_pos={0 0}, upload=507613]: Done uploading piece. err: <nil> Dec 29 18:31:55.251284281 (579h2m55.122367346s) [noise=true , address=145.79.227.46:4602 , piece_id=D77OTONDZY335R6K5RZMDEANFQP4CBE2NHXHF4KKYPDHQJ2AAF5A, seg_pos={0 0}, upload=507613]: Done uploading piece. err: <nil> Dec 29 18:31:55.259252093 (579h2m55.130335128s) [noise=true , address=145.79.227.45:4230 , piece_id=NB5F3WY5RG5ZVI4D4GXUR7VD3CYP63LRERUFS433M3NQDJZMRHHQ, seg_pos={0 0}, upload=507613]: Done uploading piece. err: <nil> Dec 29 18:31:55.262667366 (579h2m55.133750401s) [noise=true , address=145.79.227.42:4121 , piece_id=XWUGAO5O25GICVOEXPBI2THLDLZCHYVSEAXQL4S6HBVTKULAXAPA, seg_pos={0 0}, upload=507613]: Done uploading piece. err: <nil> Dec 29 18:31:55.265086382 (579h2m55.136169417s) [noise=true , address=145.79.227.38:4979 , piece_id=KYS4Q6FY5SVKBTGZCMMRGXKI6A5JBW6VT7MRCU3RVJQ3XHT4DGEQ, seg_pos={0 0}, upload=507613]: Done uploading piece. err: <nil> Dec 29 18:31:55.269663218 (579h2m55.140746253s) [noise=true , address=145.79.227.50:4375 , piece_id=5AHPGMCU7K2C63QQDG5ZAKILRHFNFT4RTWNHNOWP66PME4FFK7UA, seg_pos={0 0}, upload=507613]: Done uploading piece. err: <nil> Dec 29 18:31:55.271302910 (579h2m55.142385945s) [noise=true , address=145.79.227.12:4765 , piece_id=V366P2EAHA2PDAJNPF3DYMMS6DBXCKV53VSKH4NEUSE7VWOQV7CA, seg_pos={0 0}, upload=507613]: Done uploading piece. err: <nil> Dec 29 18:31:55.282328044 (579h2m55.153411089s) [noise=true , address=145.79.227.48:4303 , piece_id=7LSFLKWUFKD3A6LGR7THTWKTO7YSRUT7ZGRAPPSUZE3QXELSG6HQ, seg_pos={0 0}, upload=507613]: Done uploading piece. err: <nil> Dec 29 18:31:55.505041133 (579h2m55.376124168s) [noise=true , address=45.91.116.4:4939 , piece_id=EIGP3ZP53RHBVFVZPNUADEKX6S53NVO22D5SKNEQ2F6TMEXTFRJQ, seg_pos={0 0}, upload=507613]: Done uploading piece. err: <nil> Dec 29 18:31:55.524835895 (579h2m55.39591894s) [noise=true , address=77.90.31.10:4520 , piece_id=UEI5HMGDDUXHH4WXFWUBZGDR5MBD4LGBNRMB24P4TLRYNMNPJIWA, seg_pos={0 0}, upload=507613]: Done uploading piece. err: <nil> Dec 29 18:31:55.606848680 (579h2m55.477931715s) [noise=true , address=45.91.116.10:4082 , piece_id=XUQIEWBURLIUSQHIZY24OH3CYK4EDY5ZSWCJO6CM6FAEO44O7ZFA, seg_pos={0 0}, upload=507613]: Done uploading piece. err: <nil> Dec 29 18:31:55.737659594 (579h2m55.608742629s) [noise=true , address=45.91.116.12:4227 , piece_id=ZK22ZK4R32QW4KNNGDJ24HEXYUCDZYJ6TDPBQNEH7GAIGOGQKSNA, seg_pos={0 0}, upload=507613]: Done uploading piece. err: <nil> Dec 29 18:31:56.162063915 (579h2m56.03314695s) [noise=true , address=77.90.31.11:4555 , piece_id=4LW7HXKNSAPFKEXYNQF6B74Q6HJPALHI2XJ4EZPUY75XFUUED5UQ, seg_pos={0 0}, upload=507613]: Done uploading piece. err: <nil> Dec 29 18:31:56.162084584 (579h2m56.033167619s) [seg_pos={0 0}, upload=507613]: Segment reached optimal threshold of 13 pieces. Dec 29 18:31:56.162263825 (579h2m56.03334686s) [noise=true , address=145.79.227.26:4655 , piece_id=GTVOELMMLZ3CDKCPXYAO3IP3WYFYGEKW6TC5WV2BNJJWAJLKAETQ, seg_pos={0 0}, upload=507613]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:58120->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:58120->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58120->145.79.227.26:4655: use of closed network connection Dec 29 18:31:56.162269465 (579h2m56.0333525s) [noise=true , address=145.79.227.26:4655 , piece_id=GTVOELMMLZ3CDKCPXYAO3IP3WYFYGEKW6TC5WV2BNJJWAJLKAETQ, seg_pos={0 0}, upload=507613]: Long tail cancellation due to optimal threshold Dec 29 18:31:56.162487962 (579h2m56.033571017s) [noise=true , address=145.79.227.37:4565 , piece_id=WFBSXJ2KPFM4W3J23G5ZCP2TQM3VL6TE76G2BDXXTVTKLG34HJWA, seg_pos={0 0}, upload=507613]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: write tcp 188.166.225.170:46610->145.79.227.37:4565: use of closed network connection; write tcp 188.166.225.170:46610->145.79.227.37:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46610->145.79.227.37:4565: use of closed network connection Dec 29 18:31:56.162509272 (579h2m56.033592307s) [noise=true , address=145.79.227.30:4761 , piece_id=SE7OVOLW6QYVEMIT2ON63BHZ7Q4HJI5P5XZVFL2O3WQP3AONRO6A, seg_pos={0 0}, upload=507613]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 18:31:56.162517268 (579h2m56.033600313s) [noise=true , address=145.79.227.30:4761 , piece_id=SE7OVOLW6QYVEMIT2ON63BHZ7Q4HJI5P5XZVFL2O3WQP3AONRO6A, seg_pos={0 0}, upload=507613]: Long tail cancellation due to optimal threshold Dec 29 18:31:56.162524672 (579h2m56.033607707s) [noise=true , address=145.79.227.37:4565 , piece_id=WFBSXJ2KPFM4W3J23G5ZCP2TQM3VL6TE76G2BDXXTVTKLG34HJWA, seg_pos={0 0}, upload=507613]: Long tail cancellation due to optimal threshold Dec 29 18:31:56.162565460 (579h2m56.033648495s) [seg_pos={0 0}, upload=507613]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:31:56.162642326 (579h2m56.033725361s) [upload=507613]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:31:56.162649690 (579h2m56.033732735s) [upload=507613]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:31:56.448321320 (579h2m56.319404355s) [upload=507613]: Done uploading segments... Dec 29 18:31:56.448344635 (579h2m56.31942767s) [upload=507613]: Upload finished. err: <nil> Duration: 30.002472087s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 46.737402115s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:32:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 46.737421543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 34.580365475s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:31:35 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 34.58039384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.] duration: 5.749405351s trace: https://jaeger.dev.storj.io/trace/29c5c6ac90848c81 FAIL (2025-12-29 18:31:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 18:30:21.563107924 (579h1m21.434191039s) [upload=507588]: Starting upload Dec 29 18:30:21.563216872 (579h1m21.434299917s) [upload=507588]: Done starting upload Dec 29 18:30:21.563249554 (579h1m21.434332589s) [upload=507588]: Uploading segments... Dec 29 18:30:21.563526621 (579h1m21.434609677s) [upload=507588]: Got next segment. Inline: false Dec 29 18:30:21.563568151 (579h1m21.434651327s) [upload=507588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:30:21.563574143 (579h1m21.434657188s) [upload=507588]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:30:21.810983717 (579h1m21.682066772s) [seg_pos={0 0}, upload=507588]: Begin upload segment... Dec 29 18:30:21.811043200 (579h1m21.682126265s) [seg_pos={0 0}, upload=507588]: Done begin upload segment. Dec 29 18:30:21.811079119 (579h1m21.682162174s) [noise=true , address=145.79.227.42:4123 , piece_id=WR2ZUUHHRKGSKH5OQJIPNIPU4H6NB5TL44Z7ME3V6HMXZSVUR5YQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.811150525 (579h1m21.68223356s) [noise=true , address=145.79.227.51:4411 , piece_id=H7R75YDD6ZJVZUDZCPRNFYRJ6X72T4I5VDKGYGAK5RIPWBGMTN5Q, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.811933336 (579h1m21.683016371s) [noise=true , address=45.91.116.14:4192 , piece_id=RVRNQLJUXJV7C5FDAKLKY5R22IQAHTPFCOXQWDDT7GKMHNMZY5HQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.812478914 (579h1m21.683561959s) [noise=true , address=45.91.116.5:4297 , piece_id=2LEIJYUKEYBFSP4J6KT3POCSMP244EHDKFRN74GY4RQHROLOCVVQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.812988465 (579h1m21.6840715s) [noise=true , address=77.90.31.16:4735 , piece_id=HKUKJSZGCE3WT5HG6GXIIOKMSXWPSB3HPFTD7M57JA4GUSQI56VA, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.813514767 (579h1m21.684597822s) [noise=true , address=145.79.227.6:4259 , piece_id=6E44N5ZBCYFEF74ONLVSFXHV5PY5K3VRPFNABCIKVXDDHVSQOWLA, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.814272510 (579h1m21.685355555s) [noise=true , address=145.79.227.19:4077 , piece_id=EHQMGRZ6HL6INO5TEI7NZXZRNPNKXZCR47W5SJ4TXX3WVWF6NZKQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.815491341 (579h1m21.686574376s) [noise=true , address=145.79.227.49:4338 , piece_id=7NZZTTW5G32UD2HRPAMOQGBX43PRNAJSITBFP2CSJ67IWIU6Z3KQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.816181625 (579h1m21.68726467s) [noise=true , address=77.90.31.17:4773 , piece_id=VNKGGZZGWCPXG6LJJTVRNGMZ62BQJA6YDCAGV7T7C3Y2YUCWYIRQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.817219491 (579h1m21.688302526s) [noise=true , address=145.79.227.13:4803 , piece_id=4ILFX4JVDBGIKYJL6ETS5YZJNK53MBVWSKKH5NZRSWCHXG37ZDWQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.818072696 (579h1m21.689155741s) [noise=true , address=145.79.227.21:4475 , piece_id=GG5B4IO7UURQSNITVBNVA2BTTIZPC5SPPKQZPUKFY3JMWWFPTDYA, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.819082218 (579h1m21.690165253s) [noise=true , address=145.79.227.50:4375 , piece_id=SQR2M2IYCJXU4VMBVOXUWAV7WPUEZO4JPP4OVKG7GGAVORVNQ3WQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.820826720 (579h1m21.691909765s) [noise=true , address=145.79.227.32:4835 , piece_id=TRDKK3GC42A7F64R776MEU6BREEJLHZA5JHSKCX7SJGXYMRYZCFA, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.822387893 (579h1m21.693470928s) [noise=true , address=45.91.116.10:4081 , piece_id=X34SME2SQK55IILMUUMDONNR7GWDSMCUV7ELK7WH3HXPT62XIBBQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.822940074 (579h1m21.694023109s) [noise=true , address=45.91.116.4:4940 , piece_id=PYDQVGIL7UMEMFK7CCDQNYM5QC7ZJ44HPMUJLE6UY4JY3T7YSN5A, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.824990298 (579h1m21.696073343s) [noise=true , address=45.91.116.11:4120 , piece_id=3ZDIPMLP4LPN2Y3BFIITRW22AHPWSKCWGSFXIGHESFYGFUI6LDQA, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:23.524238975 (579h1m23.39532201s) [upload=507588]: Next returned nil segment Dec 29 18:30:23.524255697 (579h1m23.395338732s) [upload=507588]: Waiting for error group managing segments... Dec 29 18:30:23.808239445 (579h1m23.67932249s) [noise=true , address=145.79.227.6:4259 , piece_id=6E44N5ZBCYFEF74ONLVSFXHV5PY5K3VRPFNABCIKVXDDHVSQOWLA, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:23.810639195 (579h1m23.68172224s) [noise=true , address=145.79.227.51:4411 , piece_id=H7R75YDD6ZJVZUDZCPRNFYRJ6X72T4I5VDKGYGAK5RIPWBGMTN5Q, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:23.811023847 (579h1m23.682106882s) [noise=true , address=145.79.227.19:4077 , piece_id=EHQMGRZ6HL6INO5TEI7NZXZRNPNKXZCR47W5SJ4TXX3WVWF6NZKQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:23.820177136 (579h1m23.691260171s) [noise=true , address=145.79.227.32:4835 , piece_id=TRDKK3GC42A7F64R776MEU6BREEJLHZA5JHSKCX7SJGXYMRYZCFA, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:23.821504895 (579h1m23.69258793s) [noise=true , address=145.79.227.21:4475 , piece_id=GG5B4IO7UURQSNITVBNVA2BTTIZPC5SPPKQZPUKFY3JMWWFPTDYA, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:23.828252363 (579h1m23.699335398s) [noise=true , address=145.79.227.42:4123 , piece_id=WR2ZUUHHRKGSKH5OQJIPNIPU4H6NB5TL44Z7ME3V6HMXZSVUR5YQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:23.828673104 (579h1m23.699756139s) [noise=true , address=145.79.227.13:4803 , piece_id=4ILFX4JVDBGIKYJL6ETS5YZJNK53MBVWSKKH5NZRSWCHXG37ZDWQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:23.839117913 (579h1m23.710200948s) [noise=true , address=145.79.227.50:4375 , piece_id=SQR2M2IYCJXU4VMBVOXUWAV7WPUEZO4JPP4OVKG7GGAVORVNQ3WQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:24.118573879 (579h1m23.989656914s) [noise=true , address=77.90.31.17:4773 , piece_id=VNKGGZZGWCPXG6LJJTVRNGMZ62BQJA6YDCAGV7T7C3Y2YUCWYIRQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:24.171158719 (579h1m24.042241754s) [noise=true , address=45.91.116.14:4192 , piece_id=RVRNQLJUXJV7C5FDAKLKY5R22IQAHTPFCOXQWDDT7GKMHNMZY5HQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:24.179422855 (579h1m24.05050589s) [noise=true , address=45.91.116.4:4940 , piece_id=PYDQVGIL7UMEMFK7CCDQNYM5QC7ZJ44HPMUJLE6UY4JY3T7YSN5A, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:24.242677040 (579h1m24.113760085s) [noise=true , address=45.91.116.10:4081 , piece_id=X34SME2SQK55IILMUUMDONNR7GWDSMCUV7ELK7WH3HXPT62XIBBQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:24.696611410 (579h1m24.567694456s) [noise=true , address=77.90.31.16:4735 , piece_id=HKUKJSZGCE3WT5HG6GXIIOKMSXWPSB3HPFTD7M57JA4GUSQI56VA, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:24.696630227 (579h1m24.567713262s) [seg_pos={0 0}, upload=507588]: Segment reached optimal threshold of 13 pieces. Dec 29 18:30:24.696789970 (579h1m24.567873005s) [noise=true , address=45.91.116.5:4297 , piece_id=2LEIJYUKEYBFSP4J6KT3POCSMP244EHDKFRN74GY4RQHROLOCVVQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60822->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60822->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60822->45.91.116.5:4297: use of closed network connection Dec 29 18:30:24.696795552 (579h1m24.567878587s) [noise=true , address=45.91.116.5:4297 , piece_id=2LEIJYUKEYBFSP4J6KT3POCSMP244EHDKFRN74GY4RQHROLOCVVQ, seg_pos={0 0}, upload=507588]: Long tail cancellation due to optimal threshold Dec 29 18:30:24.696807875 (579h1m24.56789091s) [noise=true , address=45.91.116.11:4120 , piece_id=3ZDIPMLP4LPN2Y3BFIITRW22AHPWSKCWGSFXIGHESFYGFUI6LDQA, seg_pos={0 0}, upload=507588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 29 18:30:24.696816040 (579h1m24.567899075s) [noise=true , address=45.91.116.11:4120 , piece_id=3ZDIPMLP4LPN2Y3BFIITRW22AHPWSKCWGSFXIGHESFYGFUI6LDQA, seg_pos={0 0}, upload=507588]: Long tail cancellation due to optimal threshold Dec 29 18:30:24.697156258 (579h1m24.568239293s) [noise=true , address=145.79.227.49:4338 , piece_id=7NZZTTW5G32UD2HRPAMOQGBX43PRNAJSITBFP2CSJ67IWIU6Z3KQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 18:30:24.697164864 (579h1m24.56824791s) [noise=true , address=145.79.227.49:4338 , piece_id=7NZZTTW5G32UD2HRPAMOQGBX43PRNAJSITBFP2CSJ67IWIU6Z3KQ, seg_pos={0 0}, upload=507588]: Long tail cancellation due to optimal threshold Dec 29 18:30:24.697181847 (579h1m24.568264882s) [seg_pos={0 0}, upload=507588]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:30:24.697271027 (579h1m24.568354062s) [upload=507588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:30:24.697275615 (579h1m24.56835865s) [upload=507588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:30:25.033417742 (579h1m24.904500787s) [upload=507588]: Done uploading segments... Dec 29 18:30:25.033440866 (579h1m24.904523911s) [upload=507588]: Upload finished. err: <nil> Duration: 5.749435288s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.461574956s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:31:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.461607609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 19.610899625s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:30:52 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 19.610923029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X] duration: 30.00367668s trace: https://jaeger.dev.storj.io/trace/29c5c6ac90848c81 FAIL (2025-12-29 18:30:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 18:30:21.563107924 (579h1m21.434191039s) [upload=507588]: Starting upload Dec 29 18:30:21.563216872 (579h1m21.434299917s) [upload=507588]: Done starting upload Dec 29 18:30:21.563249554 (579h1m21.434332589s) [upload=507588]: Uploading segments... Dec 29 18:30:21.563526621 (579h1m21.434609677s) [upload=507588]: Got next segment. Inline: false Dec 29 18:30:21.563568151 (579h1m21.434651327s) [upload=507588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:30:21.563574143 (579h1m21.434657188s) [upload=507588]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:30:21.810983717 (579h1m21.682066772s) [seg_pos={0 0}, upload=507588]: Begin upload segment... Dec 29 18:30:21.811043200 (579h1m21.682126265s) [seg_pos={0 0}, upload=507588]: Done begin upload segment. Dec 29 18:30:21.811079119 (579h1m21.682162174s) [noise=true , address=145.79.227.42:4123 , piece_id=WR2ZUUHHRKGSKH5OQJIPNIPU4H6NB5TL44Z7ME3V6HMXZSVUR5YQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.811150525 (579h1m21.68223356s) [noise=true , address=145.79.227.51:4411 , piece_id=H7R75YDD6ZJVZUDZCPRNFYRJ6X72T4I5VDKGYGAK5RIPWBGMTN5Q, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.811933336 (579h1m21.683016371s) [noise=true , address=45.91.116.14:4192 , piece_id=RVRNQLJUXJV7C5FDAKLKY5R22IQAHTPFCOXQWDDT7GKMHNMZY5HQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.812478914 (579h1m21.683561959s) [noise=true , address=45.91.116.5:4297 , piece_id=2LEIJYUKEYBFSP4J6KT3POCSMP244EHDKFRN74GY4RQHROLOCVVQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.812988465 (579h1m21.6840715s) [noise=true , address=77.90.31.16:4735 , piece_id=HKUKJSZGCE3WT5HG6GXIIOKMSXWPSB3HPFTD7M57JA4GUSQI56VA, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.813514767 (579h1m21.684597822s) [noise=true , address=145.79.227.6:4259 , piece_id=6E44N5ZBCYFEF74ONLVSFXHV5PY5K3VRPFNABCIKVXDDHVSQOWLA, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.814272510 (579h1m21.685355555s) [noise=true , address=145.79.227.19:4077 , piece_id=EHQMGRZ6HL6INO5TEI7NZXZRNPNKXZCR47W5SJ4TXX3WVWF6NZKQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.815491341 (579h1m21.686574376s) [noise=true , address=145.79.227.49:4338 , piece_id=7NZZTTW5G32UD2HRPAMOQGBX43PRNAJSITBFP2CSJ67IWIU6Z3KQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.816181625 (579h1m21.68726467s) [noise=true , address=77.90.31.17:4773 , piece_id=VNKGGZZGWCPXG6LJJTVRNGMZ62BQJA6YDCAGV7T7C3Y2YUCWYIRQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.817219491 (579h1m21.688302526s) [noise=true , address=145.79.227.13:4803 , piece_id=4ILFX4JVDBGIKYJL6ETS5YZJNK53MBVWSKKH5NZRSWCHXG37ZDWQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.818072696 (579h1m21.689155741s) [noise=true , address=145.79.227.21:4475 , piece_id=GG5B4IO7UURQSNITVBNVA2BTTIZPC5SPPKQZPUKFY3JMWWFPTDYA, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.819082218 (579h1m21.690165253s) [noise=true , address=145.79.227.50:4375 , piece_id=SQR2M2IYCJXU4VMBVOXUWAV7WPUEZO4JPP4OVKG7GGAVORVNQ3WQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.820826720 (579h1m21.691909765s) [noise=true , address=145.79.227.32:4835 , piece_id=TRDKK3GC42A7F64R776MEU6BREEJLHZA5JHSKCX7SJGXYMRYZCFA, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.822387893 (579h1m21.693470928s) [noise=true , address=45.91.116.10:4081 , piece_id=X34SME2SQK55IILMUUMDONNR7GWDSMCUV7ELK7WH3HXPT62XIBBQ, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.822940074 (579h1m21.694023109s) [noise=true , address=45.91.116.4:4940 , piece_id=PYDQVGIL7UMEMFK7CCDQNYM5QC7ZJ44HPMUJLE6UY4JY3T7YSN5A, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:21.824990298 (579h1m21.696073343s) [noise=true , address=45.91.116.11:4120 , piece_id=3ZDIPMLP4LPN2Y3BFIITRW22AHPWSKCWGSFXIGHESFYGFUI6LDQA, seg_pos={0 0}, upload=507588]: Uploading piece... Dec 29 18:30:23.524238975 (579h1m23.39532201s) [upload=507588]: Next returned nil segment Dec 29 18:30:23.524255697 (579h1m23.395338732s) [upload=507588]: Waiting for error group managing segments... Dec 29 18:30:23.808239445 (579h1m23.67932249s) [noise=true , address=145.79.227.6:4259 , piece_id=6E44N5ZBCYFEF74ONLVSFXHV5PY5K3VRPFNABCIKVXDDHVSQOWLA, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:23.810639195 (579h1m23.68172224s) [noise=true , address=145.79.227.51:4411 , piece_id=H7R75YDD6ZJVZUDZCPRNFYRJ6X72T4I5VDKGYGAK5RIPWBGMTN5Q, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:23.811023847 (579h1m23.682106882s) [noise=true , address=145.79.227.19:4077 , piece_id=EHQMGRZ6HL6INO5TEI7NZXZRNPNKXZCR47W5SJ4TXX3WVWF6NZKQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:23.820177136 (579h1m23.691260171s) [noise=true , address=145.79.227.32:4835 , piece_id=TRDKK3GC42A7F64R776MEU6BREEJLHZA5JHSKCX7SJGXYMRYZCFA, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:23.821504895 (579h1m23.69258793s) [noise=true , address=145.79.227.21:4475 , piece_id=GG5B4IO7UURQSNITVBNVA2BTTIZPC5SPPKQZPUKFY3JMWWFPTDYA, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:23.828252363 (579h1m23.699335398s) [noise=true , address=145.79.227.42:4123 , piece_id=WR2ZUUHHRKGSKH5OQJIPNIPU4H6NB5TL44Z7ME3V6HMXZSVUR5YQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:23.828673104 (579h1m23.699756139s) [noise=true , address=145.79.227.13:4803 , piece_id=4ILFX4JVDBGIKYJL6ETS5YZJNK53MBVWSKKH5NZRSWCHXG37ZDWQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:23.839117913 (579h1m23.710200948s) [noise=true , address=145.79.227.50:4375 , piece_id=SQR2M2IYCJXU4VMBVOXUWAV7WPUEZO4JPP4OVKG7GGAVORVNQ3WQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:24.118573879 (579h1m23.989656914s) [noise=true , address=77.90.31.17:4773 , piece_id=VNKGGZZGWCPXG6LJJTVRNGMZ62BQJA6YDCAGV7T7C3Y2YUCWYIRQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:24.171158719 (579h1m24.042241754s) [noise=true , address=45.91.116.14:4192 , piece_id=RVRNQLJUXJV7C5FDAKLKY5R22IQAHTPFCOXQWDDT7GKMHNMZY5HQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:24.179422855 (579h1m24.05050589s) [noise=true , address=45.91.116.4:4940 , piece_id=PYDQVGIL7UMEMFK7CCDQNYM5QC7ZJ44HPMUJLE6UY4JY3T7YSN5A, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:24.242677040 (579h1m24.113760085s) [noise=true , address=45.91.116.10:4081 , piece_id=X34SME2SQK55IILMUUMDONNR7GWDSMCUV7ELK7WH3HXPT62XIBBQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:24.696611410 (579h1m24.567694456s) [noise=true , address=77.90.31.16:4735 , piece_id=HKUKJSZGCE3WT5HG6GXIIOKMSXWPSB3HPFTD7M57JA4GUSQI56VA, seg_pos={0 0}, upload=507588]: Done uploading piece. err: <nil> Dec 29 18:30:24.696630227 (579h1m24.567713262s) [seg_pos={0 0}, upload=507588]: Segment reached optimal threshold of 13 pieces. Dec 29 18:30:24.696789970 (579h1m24.567873005s) [noise=true , address=45.91.116.5:4297 , piece_id=2LEIJYUKEYBFSP4J6KT3POCSMP244EHDKFRN74GY4RQHROLOCVVQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:60822->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:60822->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60822->45.91.116.5:4297: use of closed network connection Dec 29 18:30:24.696795552 (579h1m24.567878587s) [noise=true , address=45.91.116.5:4297 , piece_id=2LEIJYUKEYBFSP4J6KT3POCSMP244EHDKFRN74GY4RQHROLOCVVQ, seg_pos={0 0}, upload=507588]: Long tail cancellation due to optimal threshold Dec 29 18:30:24.696807875 (579h1m24.56789091s) [noise=true , address=45.91.116.11:4120 , piece_id=3ZDIPMLP4LPN2Y3BFIITRW22AHPWSKCWGSFXIGHESFYGFUI6LDQA, seg_pos={0 0}, upload=507588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 29 18:30:24.696816040 (579h1m24.567899075s) [noise=true , address=45.91.116.11:4120 , piece_id=3ZDIPMLP4LPN2Y3BFIITRW22AHPWSKCWGSFXIGHESFYGFUI6LDQA, seg_pos={0 0}, upload=507588]: Long tail cancellation due to optimal threshold Dec 29 18:30:24.697156258 (579h1m24.568239293s) [noise=true , address=145.79.227.49:4338 , piece_id=7NZZTTW5G32UD2HRPAMOQGBX43PRNAJSITBFP2CSJ67IWIU6Z3KQ, seg_pos={0 0}, upload=507588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 18:30:24.697164864 (579h1m24.56824791s) [noise=true , address=145.79.227.49:4338 , piece_id=7NZZTTW5G32UD2HRPAMOQGBX43PRNAJSITBFP2CSJ67IWIU6Z3KQ, seg_pos={0 0}, upload=507588]: Long tail cancellation due to optimal threshold Dec 29 18:30:24.697181847 (579h1m24.568264882s) [seg_pos={0 0}, upload=507588]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:30:24.697271027 (579h1m24.568354062s) [upload=507588]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:30:24.697275615 (579h1m24.56835865s) [upload=507588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:30:25.033417742 (579h1m24.904500787s) [upload=507588]: Done uploading segments... Dec 29 18:30:25.033440866 (579h1m24.904523911s) [upload=507588]: Upload finished. err: <nil> Duration: 30.003698922s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.512147233s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:30:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.512180786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 57.448381341s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:30:29 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 57.448400417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.] duration: 7.846267007s trace: https://jaeger.dev.storj.io/trace/4792a504f916a2ba FAIL (2025-12-29 18:29:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: XPXPE2AM345CZBP6RMWCHUPELHZSRFBHNWXP75C4CADNJQL2T4JQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: OILHF4XPAS3QMYKLCOIH6WXGDUGAYK64ABS7VCIZWSLTNRC7ASDA) context deadline exceeded ======LOGS====== 28:40.887870949 (578h59m40.758953984s) [upload=507562]: Starting upload Dec 29 18:28:40.887922718 (578h59m40.759005753s) [upload=507562]: Done starting upload Dec 29 18:28:40.888041083 (578h59m40.759124118s) [upload=507562]: Uploading segments... Dec 29 18:28:40.888182052 (578h59m40.759265097s) [upload=507562]: Got next segment. Inline: false Dec 29 18:28:40.888210997 (578h59m40.759294032s) [upload=507562]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:28:40.888216187 (578h59m40.759299222s) [upload=507562]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:28:41.132250546 (578h59m41.003333591s) [seg_pos={0 0}, upload=507562]: Begin upload segment... Dec 29 18:28:41.132298737 (578h59m41.003381772s) [seg_pos={0 0}, upload=507562]: Done begin upload segment. Dec 29 18:28:41.132316771 (578h59m41.003399806s) [noise=true , address=77.90.31.18:4805 , piece_id=OILHF4XPAS3QMYKLCOIH6WXGDUGAYK64ABS7VCIZWSLTNRC7ASDA, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.132365945 (578h59m41.00344898s) [noise=true , address=145.79.227.37:4566 , piece_id=XU2VQNT2TFB6YPFEEGFBMZXEHDBP2RRWUNEZJX7IX424YGSKJRYQ, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.133012656 (578h59m41.004095701s) [noise=true , address=145.79.227.11:4729 , piece_id=JRD6FWMUFSNS6TBYYVGDCFQOH6YXLUVGE2Y6FWYU3564XLOY7XBQ, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.133298842 (578h59m41.004381877s) [noise=true , address=145.79.227.45:4231 , piece_id=WXKVS3TEISGOMESDXSC6CZCTVGQ2BAPZJWJOM7F4MHSI5FELO6WA, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.133569276 (578h59m41.004652311s) [noise=true , address=145.79.227.46:4603 , piece_id=OEQEFEPOL766E2NC3FWVMLBBUO3Z2JYJ2O5SUVPS6ITZ5YO7N4NQ, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.134004575 (578h59m41.00508761s) [noise=true , address=145.79.227.17:4972 , piece_id=VBFXUBS7AGANNSFYRZOJYIB6V4UWFSDRCMHH3PL6PFL34C6HEJPQ, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.134315678 (578h59m41.005398723s) [noise=true , address=77.90.31.13:4626 , piece_id=W42CHR7I5TI4JLUIWO4UH5A32BQ4NCZQRXB7EMQGB2SWMLPODWPA, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.134595931 (578h59m41.005678967s) [noise=true , address=145.79.227.22:4509 , piece_id=CCJUVKJBJXTKCDGLGUINO65TFJLX5LVQ3XWL3XIB7RHKM3H3JTXA, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.134997496 (578h59m41.006080531s) [noise=true , address=145.79.227.42:4121 , piece_id=VDYOKFK7O5ZEI66H6RJIAKUX2LXELMKMKUHQ7AKBIW7UDANX3CMA, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.136059589 (578h59m41.007142634s) [noise=true , address=45.91.116.12:4227 , piece_id=VX3WFZ4LRQ46RVIMHCCCW6EBGHFQ2YWOBN4UCA3PRDO7WGPHVUWA, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.136204043 (578h59m41.007287119s) [noise=true , address=145.79.227.36:4518 , piece_id=CWHCAEEML6OEKYXEALJ4VDH6LOYBDIX463NQHPSFWXK5BO4R32LQ, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.137253191 (578h59m41.008336236s) [noise=true , address=77.90.31.14:4661 , piece_id=XPXPE2AM345CZBP6RMWCHUPELHZSRFBHNWXP75C4CADNJQL2T4JQ, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.137376025 (578h59m41.00845906s) [noise=true , address=45.91.116.11:4117 , piece_id=IK2WEUCEFJFH6R7MNF36V3JFXRLMPZZV5QC4TENPCXFQS4WFRFOQ, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.138448557 (578h59m41.009531602s) [noise=true , address=145.79.227.38:4978 , piece_id=65QKPILJ34LYQ7Z5F4LZPUXU2IKZIS5JOBRFHKAUXL4XDY4W6DDA, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.138822810 (578h59m41.009905855s) [noise=true , address=145.79.227.47:4267 , piece_id=EP4TU5IA5QJD6GIADWZUNHOYQHN4OVFZMXYMOC7CRVN2CSSYIVYA, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.139578269 (578h59m41.010661304s) [noise=true , address=145.79.227.2:4115 , piece_id=WPKH5SFNR4A3ADWNIWQP5AUJYYDQFHCKLDYJEC5P4BW5AOWZWVXQ, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:42.827456987 (578h59m42.698540032s) [upload=507562]: Next returned nil segment Dec 29 18:28:42.827468459 (578h59m42.698551504s) [upload=507562]: Waiting for error group managing segments... Dec 29 18:28:43.109394179 (578h59m42.980477214s) [noise=true , address=145.79.227.45:4231 , piece_id=WXKVS3TEISGOMESDXSC6CZCTVGQ2BAPZJWJOM7F4MHSI5FELO6WA, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.118286982 (578h59m42.989370017s) [noise=true , address=145.79.227.46:4603 , piece_id=OEQEFEPOL766E2NC3FWVMLBBUO3Z2JYJ2O5SUVPS6ITZ5YO7N4NQ, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.127475899 (578h59m42.998558934s) [noise=true , address=145.79.227.11:4729 , piece_id=JRD6FWMUFSNS6TBYYVGDCFQOH6YXLUVGE2Y6FWYU3564XLOY7XBQ, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.130340924 (578h59m43.001423969s) [noise=true , address=145.79.227.38:4978 , piece_id=65QKPILJ34LYQ7Z5F4LZPUXU2IKZIS5JOBRFHKAUXL4XDY4W6DDA, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.136173680 (578h59m43.007256715s) [noise=true , address=145.79.227.37:4566 , piece_id=XU2VQNT2TFB6YPFEEGFBMZXEHDBP2RRWUNEZJX7IX424YGSKJRYQ, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.139849380 (578h59m43.010932425s) [noise=true , address=145.79.227.47:4267 , piece_id=EP4TU5IA5QJD6GIADWZUNHOYQHN4OVFZMXYMOC7CRVN2CSSYIVYA, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.144684266 (578h59m43.015767312s) [noise=true , address=145.79.227.36:4518 , piece_id=CWHCAEEML6OEKYXEALJ4VDH6LOYBDIX463NQHPSFWXK5BO4R32LQ, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.155158792 (578h59m43.026241837s) [noise=true , address=145.79.227.42:4121 , piece_id=VDYOKFK7O5ZEI66H6RJIAKUX2LXELMKMKUHQ7AKBIW7UDANX3CMA, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.159064288 (578h59m43.030147323s) [noise=true , address=145.79.227.2:4115 , piece_id=WPKH5SFNR4A3ADWNIWQP5AUJYYDQFHCKLDYJEC5P4BW5AOWZWVXQ, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.201743665 (578h59m43.07282671s) [noise=true , address=145.79.227.22:4509 , piece_id=CCJUVKJBJXTKCDGLGUINO65TFJLX5LVQ3XWL3XIB7RHKM3H3JTXA, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.312949869 (578h59m43.184032905s) [noise=true , address=77.90.31.14:4661 , piece_id=XPXPE2AM345CZBP6RMWCHUPELHZSRFBHNWXP75C4CADNJQL2T4JQ, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.393406702 (578h59m43.264489737s) [noise=true , address=77.90.31.18:4805 , piece_id=OILHF4XPAS3QMYKLCOIH6WXGDUGAYK64ABS7VCIZWSLTNRC7ASDA, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.443067583 (578h59m43.314150618s) [noise=true , address=45.91.116.11:4117 , piece_id=IK2WEUCEFJFH6R7MNF36V3JFXRLMPZZV5QC4TENPCXFQS4WFRFOQ, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.443087231 (578h59m43.314170266s) [seg_pos={0 0}, upload=507562]: Segment reached optimal threshold of 13 pieces. Dec 29 18:28:43.443270229 (578h59m43.314353264s) [noise=true , address=45.91.116.12:4227 , piece_id=VX3WFZ4LRQ46RVIMHCCCW6EBGHFQ2YWOBN4UCA3PRDO7WGPHVUWA, seg_pos={0 0}, upload=507562]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 29 18:28:43.443276992 (578h59m43.314360027s) [noise=true , address=45.91.116.12:4227 , piece_id=VX3WFZ4LRQ46RVIMHCCCW6EBGHFQ2YWOBN4UCA3PRDO7WGPHVUWA, seg_pos={0 0}, upload=507562]: Long tail cancellation due to optimal threshold Dec 29 18:28:43.443472795 (578h59m43.31455583s) [noise=true , address=145.79.227.17:4972 , piece_id=VBFXUBS7AGANNSFYRZOJYIB6V4UWFSDRCMHH3PL6PFL34C6HEJPQ, seg_pos={0 0}, upload=507562]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:52774->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52774->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52774->145.79.227.17:4972: use of closed network connection Dec 29 18:28:43.443479498 (578h59m43.314562533s) [noise=true , address=145.79.227.17:4972 , piece_id=VBFXUBS7AGANNSFYRZOJYIB6V4UWFSDRCMHH3PL6PFL34C6HEJPQ, seg_pos={0 0}, upload=507562]: Long tail cancellation due to optimal threshold Dec 29 18:28:43.443752307 (578h59m43.314835342s) [noise=true , address=77.90.31.13:4626 , piece_id=W42CHR7I5TI4JLUIWO4UH5A32BQ4NCZQRXB7EMQGB2SWMLPODWPA, seg_pos={0 0}, upload=507562]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 29 18:28:43.443770191 (578h59m43.314853236s) [noise=true , address=77.90.31.13:4626 , piece_id=W42CHR7I5TI4JLUIWO4UH5A32BQ4NCZQRXB7EMQGB2SWMLPODWPA, seg_pos={0 0}, upload=507562]: Long tail cancellation due to optimal threshold Dec 29 18:28:43.443813524 (578h59m43.314896559s) [seg_pos={0 0}, upload=507562]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:28:43.443931749 (578h59m43.315014784s) [upload=507562]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:28:43.443938752 (578h59m43.315021787s) [upload=507562]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:28:43.733529186 (578h59m43.604612231s) [upload=507562]: Done uploading segments... Dec 29 18:28:43.733548333 (578h59m43.604631368s) [upload=507562]: Upload finished. err: <nil> Duration: 7.846291706s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 40.083783856s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:29:32 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 40.08381178s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.509028361s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:29:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.509057065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX] duration: 30.003360184s trace: https://jaeger.dev.storj.io/trace/4792a504f916a2ba FAIL (2025-12-29 18:29:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: XPXPE2AM345CZBP6RMWCHUPELHZSRFBHNWXP75C4CADNJQL2T4JQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: OILHF4XPAS3QMYKLCOIH6WXGDUGAYK64ABS7VCIZWSLTNRC7ASDA) context deadline exceeded ======LOGS====== 28:40.887870949 (578h59m40.758953984s) [upload=507562]: Starting upload Dec 29 18:28:40.887922718 (578h59m40.759005753s) [upload=507562]: Done starting upload Dec 29 18:28:40.888041083 (578h59m40.759124118s) [upload=507562]: Uploading segments... Dec 29 18:28:40.888182052 (578h59m40.759265097s) [upload=507562]: Got next segment. Inline: false Dec 29 18:28:40.888210997 (578h59m40.759294032s) [upload=507562]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:28:40.888216187 (578h59m40.759299222s) [upload=507562]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:28:41.132250546 (578h59m41.003333591s) [seg_pos={0 0}, upload=507562]: Begin upload segment... Dec 29 18:28:41.132298737 (578h59m41.003381772s) [seg_pos={0 0}, upload=507562]: Done begin upload segment. Dec 29 18:28:41.132316771 (578h59m41.003399806s) [noise=true , address=77.90.31.18:4805 , piece_id=OILHF4XPAS3QMYKLCOIH6WXGDUGAYK64ABS7VCIZWSLTNRC7ASDA, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.132365945 (578h59m41.00344898s) [noise=true , address=145.79.227.37:4566 , piece_id=XU2VQNT2TFB6YPFEEGFBMZXEHDBP2RRWUNEZJX7IX424YGSKJRYQ, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.133012656 (578h59m41.004095701s) [noise=true , address=145.79.227.11:4729 , piece_id=JRD6FWMUFSNS6TBYYVGDCFQOH6YXLUVGE2Y6FWYU3564XLOY7XBQ, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.133298842 (578h59m41.004381877s) [noise=true , address=145.79.227.45:4231 , piece_id=WXKVS3TEISGOMESDXSC6CZCTVGQ2BAPZJWJOM7F4MHSI5FELO6WA, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.133569276 (578h59m41.004652311s) [noise=true , address=145.79.227.46:4603 , piece_id=OEQEFEPOL766E2NC3FWVMLBBUO3Z2JYJ2O5SUVPS6ITZ5YO7N4NQ, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.134004575 (578h59m41.00508761s) [noise=true , address=145.79.227.17:4972 , piece_id=VBFXUBS7AGANNSFYRZOJYIB6V4UWFSDRCMHH3PL6PFL34C6HEJPQ, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.134315678 (578h59m41.005398723s) [noise=true , address=77.90.31.13:4626 , piece_id=W42CHR7I5TI4JLUIWO4UH5A32BQ4NCZQRXB7EMQGB2SWMLPODWPA, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.134595931 (578h59m41.005678967s) [noise=true , address=145.79.227.22:4509 , piece_id=CCJUVKJBJXTKCDGLGUINO65TFJLX5LVQ3XWL3XIB7RHKM3H3JTXA, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.134997496 (578h59m41.006080531s) [noise=true , address=145.79.227.42:4121 , piece_id=VDYOKFK7O5ZEI66H6RJIAKUX2LXELMKMKUHQ7AKBIW7UDANX3CMA, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.136059589 (578h59m41.007142634s) [noise=true , address=45.91.116.12:4227 , piece_id=VX3WFZ4LRQ46RVIMHCCCW6EBGHFQ2YWOBN4UCA3PRDO7WGPHVUWA, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.136204043 (578h59m41.007287119s) [noise=true , address=145.79.227.36:4518 , piece_id=CWHCAEEML6OEKYXEALJ4VDH6LOYBDIX463NQHPSFWXK5BO4R32LQ, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.137253191 (578h59m41.008336236s) [noise=true , address=77.90.31.14:4661 , piece_id=XPXPE2AM345CZBP6RMWCHUPELHZSRFBHNWXP75C4CADNJQL2T4JQ, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.137376025 (578h59m41.00845906s) [noise=true , address=45.91.116.11:4117 , piece_id=IK2WEUCEFJFH6R7MNF36V3JFXRLMPZZV5QC4TENPCXFQS4WFRFOQ, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.138448557 (578h59m41.009531602s) [noise=true , address=145.79.227.38:4978 , piece_id=65QKPILJ34LYQ7Z5F4LZPUXU2IKZIS5JOBRFHKAUXL4XDY4W6DDA, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.138822810 (578h59m41.009905855s) [noise=true , address=145.79.227.47:4267 , piece_id=EP4TU5IA5QJD6GIADWZUNHOYQHN4OVFZMXYMOC7CRVN2CSSYIVYA, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:41.139578269 (578h59m41.010661304s) [noise=true , address=145.79.227.2:4115 , piece_id=WPKH5SFNR4A3ADWNIWQP5AUJYYDQFHCKLDYJEC5P4BW5AOWZWVXQ, seg_pos={0 0}, upload=507562]: Uploading piece... Dec 29 18:28:42.827456987 (578h59m42.698540032s) [upload=507562]: Next returned nil segment Dec 29 18:28:42.827468459 (578h59m42.698551504s) [upload=507562]: Waiting for error group managing segments... Dec 29 18:28:43.109394179 (578h59m42.980477214s) [noise=true , address=145.79.227.45:4231 , piece_id=WXKVS3TEISGOMESDXSC6CZCTVGQ2BAPZJWJOM7F4MHSI5FELO6WA, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.118286982 (578h59m42.989370017s) [noise=true , address=145.79.227.46:4603 , piece_id=OEQEFEPOL766E2NC3FWVMLBBUO3Z2JYJ2O5SUVPS6ITZ5YO7N4NQ, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.127475899 (578h59m42.998558934s) [noise=true , address=145.79.227.11:4729 , piece_id=JRD6FWMUFSNS6TBYYVGDCFQOH6YXLUVGE2Y6FWYU3564XLOY7XBQ, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.130340924 (578h59m43.001423969s) [noise=true , address=145.79.227.38:4978 , piece_id=65QKPILJ34LYQ7Z5F4LZPUXU2IKZIS5JOBRFHKAUXL4XDY4W6DDA, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.136173680 (578h59m43.007256715s) [noise=true , address=145.79.227.37:4566 , piece_id=XU2VQNT2TFB6YPFEEGFBMZXEHDBP2RRWUNEZJX7IX424YGSKJRYQ, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.139849380 (578h59m43.010932425s) [noise=true , address=145.79.227.47:4267 , piece_id=EP4TU5IA5QJD6GIADWZUNHOYQHN4OVFZMXYMOC7CRVN2CSSYIVYA, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.144684266 (578h59m43.015767312s) [noise=true , address=145.79.227.36:4518 , piece_id=CWHCAEEML6OEKYXEALJ4VDH6LOYBDIX463NQHPSFWXK5BO4R32LQ, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.155158792 (578h59m43.026241837s) [noise=true , address=145.79.227.42:4121 , piece_id=VDYOKFK7O5ZEI66H6RJIAKUX2LXELMKMKUHQ7AKBIW7UDANX3CMA, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.159064288 (578h59m43.030147323s) [noise=true , address=145.79.227.2:4115 , piece_id=WPKH5SFNR4A3ADWNIWQP5AUJYYDQFHCKLDYJEC5P4BW5AOWZWVXQ, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.201743665 (578h59m43.07282671s) [noise=true , address=145.79.227.22:4509 , piece_id=CCJUVKJBJXTKCDGLGUINO65TFJLX5LVQ3XWL3XIB7RHKM3H3JTXA, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.312949869 (578h59m43.184032905s) [noise=true , address=77.90.31.14:4661 , piece_id=XPXPE2AM345CZBP6RMWCHUPELHZSRFBHNWXP75C4CADNJQL2T4JQ, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.393406702 (578h59m43.264489737s) [noise=true , address=77.90.31.18:4805 , piece_id=OILHF4XPAS3QMYKLCOIH6WXGDUGAYK64ABS7VCIZWSLTNRC7ASDA, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.443067583 (578h59m43.314150618s) [noise=true , address=45.91.116.11:4117 , piece_id=IK2WEUCEFJFH6R7MNF36V3JFXRLMPZZV5QC4TENPCXFQS4WFRFOQ, seg_pos={0 0}, upload=507562]: Done uploading piece. err: <nil> Dec 29 18:28:43.443087231 (578h59m43.314170266s) [seg_pos={0 0}, upload=507562]: Segment reached optimal threshold of 13 pieces. Dec 29 18:28:43.443270229 (578h59m43.314353264s) [noise=true , address=45.91.116.12:4227 , piece_id=VX3WFZ4LRQ46RVIMHCCCW6EBGHFQ2YWOBN4UCA3PRDO7WGPHVUWA, seg_pos={0 0}, upload=507562]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 29 18:28:43.443276992 (578h59m43.314360027s) [noise=true , address=45.91.116.12:4227 , piece_id=VX3WFZ4LRQ46RVIMHCCCW6EBGHFQ2YWOBN4UCA3PRDO7WGPHVUWA, seg_pos={0 0}, upload=507562]: Long tail cancellation due to optimal threshold Dec 29 18:28:43.443472795 (578h59m43.31455583s) [noise=true , address=145.79.227.17:4972 , piece_id=VBFXUBS7AGANNSFYRZOJYIB6V4UWFSDRCMHH3PL6PFL34C6HEJPQ, seg_pos={0 0}, upload=507562]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:52774->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52774->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52774->145.79.227.17:4972: use of closed network connection Dec 29 18:28:43.443479498 (578h59m43.314562533s) [noise=true , address=145.79.227.17:4972 , piece_id=VBFXUBS7AGANNSFYRZOJYIB6V4UWFSDRCMHH3PL6PFL34C6HEJPQ, seg_pos={0 0}, upload=507562]: Long tail cancellation due to optimal threshold Dec 29 18:28:43.443752307 (578h59m43.314835342s) [noise=true , address=77.90.31.13:4626 , piece_id=W42CHR7I5TI4JLUIWO4UH5A32BQ4NCZQRXB7EMQGB2SWMLPODWPA, seg_pos={0 0}, upload=507562]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 29 18:28:43.443770191 (578h59m43.314853236s) [noise=true , address=77.90.31.13:4626 , piece_id=W42CHR7I5TI4JLUIWO4UH5A32BQ4NCZQRXB7EMQGB2SWMLPODWPA, seg_pos={0 0}, upload=507562]: Long tail cancellation due to optimal threshold Dec 29 18:28:43.443813524 (578h59m43.314896559s) [seg_pos={0 0}, upload=507562]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:28:43.443931749 (578h59m43.315014784s) [upload=507562]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:28:43.443938752 (578h59m43.315021787s) [upload=507562]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:28:43.733529186 (578h59m43.604612231s) [upload=507562]: Done uploading segments... Dec 29 18:28:43.733548333 (578h59m43.604631368s) [upload=507562]: Upload finished. err: <nil> Duration: 30.003388608s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.550978833s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:29:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.551519364s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X......XXX.X] duration: 30.003838323s trace: https://jaeger.dev.storj.io/trace/55ea195b57e1f9fb FAIL (2025-12-29 18:28:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 18:28:10.639840647 (578h59m10.510923702s) [upload=507556]: Starting upload Dec 29 18:28:10.639934766 (578h59m10.511017801s) [upload=507556]: Done starting upload Dec 29 18:28:10.639995011 (578h59m10.511078046s) [upload=507556]: Uploading segments... Dec 29 18:28:10.640128555 (578h59m10.5112116s) [upload=507556]: Got next segment. Inline: false Dec 29 18:28:10.640160385 (578h59m10.51124342s) [upload=507556]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:28:10.640164875 (578h59m10.51124792s) [upload=507556]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:28:10.888121070 (578h59m10.759204165s) [seg_pos={0 0}, upload=507556]: Begin upload segment... Dec 29 18:28:10.888181505 (578h59m10.75926455s) [seg_pos={0 0}, upload=507556]: Done begin upload segment. Dec 29 18:28:10.888211022 (578h59m10.759294057s) [noise=true , address=145.79.227.11:4731 , piece_id=GMNTHRY47HPBM4U5ER7JPIRN7GHCOBAW6UXHZS3JJLBB5D2DWSUQ, seg_pos={0 0}, upload=507556]: Uploading piece... Dec 29 18:28:10.888292667 (578h59m10.759375712s) [noise=true , address=145.79.227.8:4329 , piece_id=QGDTGZBVOHC6EFKJVTU3PR5COKSWOZ7NZCHLECMSDOBCTBQ3B27Q, seg_pos={0 0}, upload=507556]: Uploading piece... Dec 29 18:28:10.889044870 (578h59m10.760127915s) [noise=true , address=145.79.227.41:4086 , piece_id=6LV6JZJ6XJQHGUJXJTKE5C6LYOJSL6AZWHBCVOKFYOHUGHO5PSUA, seg_pos={0 0}, upload=507556]: Uploading piece... Dec 29 18:28:10.889531857 (578h59m10.760614902s) [noise=true , address=77.90.31.12:4589 , piece_id=WALPM7NDGGIEDUUIOKAD75JMQXIS7IFG6Z34LLJBR52KRKYRMEJA, seg_pos={0 0}, upload=507556]: Uploading piece... Dec 29 18:28:10.890095270 (578h59m10.761178315s) [noise=true , address=145.79.227.52:4446 , piece_id=A7UCPYY2MW5YC3YL3QMUXWVFKLVT7HNCO4L6YJVXI2I2L5W7KEVA, seg_pos={0 0}, upload=507556]: Uploading piece... Dec 29 18:28:10.890524677 (578h59m10.761607722s) [noise=true , address=145.79.227.39:4014 , piece_id=G4BT3UDZUKB4GV3JURVRKDCPOZGEONVM4AIOTN6AQQZRONDAI3CQ, seg_pos={0 0}, upload=507556]: Uploading piece... Dec 29 18:28:10.890978671 (578h59m10.762061716s) [noise=true , address=145.79.227.38:4977 , piece_id=PGY7ZB63AARE2N6VYWBXEFZ5FRCL6W2TBDQUUDKXI6RNATXIBPEA, seg_pos={0 0}, upload=507556]: Uploading piece... Dec 29 18:28:10.892055712 (578h59m10.763138757s) [noise=true , address=145.79.227.4:4187 , piece_id=6JLWC6M5EB5JABMF6OZF2ISJFEJJMORKUSHK226KL5IVGL2G6FDA, seg_pos={0 0}, upload=507556]: Uploading piece... Dec 29 18:28:10.892581063 (578h59m10.763664108s) [noise=true , address=145.79.227.43:4159 , piece_id=6WYIZL7I3OT22XWKEIFPXLXEPQXR3BP3W4LPOEB4D6WZ7WHTG66A, seg_pos={0 0}, upload=507556]: Uploading piece... Dec 29 18:28:10.893342623 (578h59m10.764425668s) [noise=true , address=145.79.227.15:4875 , piece_id=VELBMBY6UCYB5MI6WV6OZAR77QT3ZDHCSINANT5JALFN4HSMH4IA, seg_pos={0 0}, upload=507556]: Uploading piece... Dec 29 18:28:10.894388655 (578h59m10.7654717s) [noise=true , address=145.79.227.24:4581 , piece_id=WTQECKALHHZHR27BBBDUTA7FAKYXMAXWFR765FUYHI7PXSAUBSXQ, seg_pos={0 0}, upload=507556]: Uploading piece... Dec 29 18:28:10.894768157 (578h59m10.765851202s) [noise=true , address=145.79.227.28:4727 , piece_id=IXI4LZGF5H74SR7TFPBYFN6YTHBAS7EWWJ7HX5JGJAHHS7WMRBLQ, seg_pos={0 0}, upload=507556]: Uploading piece... Dec 29 18:28:10.895898450 (578h59m10.766981485s) [noise=true , address=145.79.227.21:4475 , piece_id=FVBQVCQ6V44VGZOQDHWGYIL3CTQDGJMOYTRLOXSBY3DOAPH272KA, seg_pos={0 0}, upload=507556]: Uploading piece... Dec 29 18:28:10.897268769 (578h59m10.768351814s) [noise=true , address=45.91.116.5:4300 , piece_id=GNZTFH554AU37PQG7MS3JZ3W5ZZD6M5APBFRTTY3YYAIG2GSTLXA, seg_pos={0 0}, upload=507556]: Uploading piece... Dec 29 18:28:10.898676269 (578h59m10.769759314s) [noise=true , address=45.91.116.12:4228 , piece_id=RVAXIMCSJBQNVMZFR4ZJS3OHV23V5S4RE6IACHR47YELFCJJXC2A, seg_pos={0 0}, upload=507556]: Uploading piece... Dec 29 18:28:10.900324157 (578h59m10.771407192s) [noise=true , address=77.90.31.11:4555 , piece_id=MUIQQLAPN3BS2J6TCVCHR3HWWMJGEJ2NUY4DAANCIAJIBWGZJIQA, seg_pos={0 0}, upload=507556]: Uploading piece... Dec 29 18:28:12.482700272 (578h59m12.353783317s) [upload=507556]: Next returned nil segment Dec 29 18:28:12.482744075 (578h59m12.35382712s) [upload=507556]: Waiting for error group managing segments... Dec 29 18:28:12.870594802 (578h59m12.741677837s) [noise=true , address=145.79.227.24:4581 , piece_id=WTQECKALHHZHR27BBBDUTA7FAKYXMAXWFR765FUYHI7PXSAUBSXQ, seg_pos={0 0}, upload=507556]: Done uploading piece. err: <nil> Dec 29 18:28:12.886457929 (578h59m12.757540974s) [noise=true , address=145.79.227.39:4014 , piece_id=G4BT3UDZUKB4GV3JURVRKDCPOZGEONVM4AIOTN6AQQZRONDAI3CQ, seg_pos={0 0}, upload=507556]: Done uploading piece. err: <nil> Dec 29 18:28:12.886789630 (578h59m12.757872675s) [noise=true , address=145.79.227.21:4475 , piece_id=FVBQVCQ6V44VGZOQDHWGYIL3CTQDGJMOYTRLOXSBY3DOAPH272KA, seg_pos={0 0}, upload=507556]: Done uploading piece. err: <nil> Dec 29 18:28:12.886962098 (578h59m12.758045143s) [noise=true , address=145.79.227.11:4731 , piece_id=GMNTHRY47HPBM4U5ER7JPIRN7GHCOBAW6UXHZS3JJLBB5D2DWSUQ, seg_pos={0 0}, upload=507556]: Done uploading piece. err: <nil> Dec 29 18:28:12.889102905 (578h59m12.76018594s) [noise=true , address=145.79.227.4:4187 , piece_id=6JLWC6M5EB5JABMF6OZF2ISJFEJJMORKUSHK226KL5IVGL2G6FDA, seg_pos={0 0}, upload=507556]: Done uploading piece. err: <nil> Dec 29 18:28:12.890111846 (578h59m12.761194882s) [noise=true , address=145.79.227.38:4977 , piece_id=PGY7ZB63AARE2N6VYWBXEFZ5FRCL6W2TBDQUUDKXI6RNATXIBPEA, seg_pos={0 0}, upload=507556]: Done uploading piece. err: <nil> Dec 29 18:28:12.896554774 (578h59m12.767637809s) [noise=true , address=145.79.227.8:4329 , piece_id=QGDTGZBVOHC6EFKJVTU3PR5COKSWOZ7NZCHLECMSDOBCTBQ3B27Q, seg_pos={0 0}, upload=507556]: Done uploading piece. err: <nil> Dec 29 18:28:12.896641349 (578h59m12.767724394s) [noise=true , address=145.79.227.28:4727 , piece_id=IXI4LZGF5H74SR7TFPBYFN6YTHBAS7EWWJ7HX5JGJAHHS7WMRBLQ, seg_pos={0 0}, upload=507556]: Done uploading piece. err: <nil> Dec 29 18:28:12.911433576 (578h59m12.782516621s) [noise=true , address=145.79.227.15:4875 , piece_id=VELBMBY6UCYB5MI6WV6OZAR77QT3ZDHCSINANT5JALFN4HSMH4IA, seg_pos={0 0}, upload=507556]: Done uploading piece. err: <nil> Dec 29 18:28:13.187787025 (578h59m13.05887007s) [noise=true , address=77.90.31.12:4589 , piece_id=WALPM7NDGGIEDUUIOKAD75JMQXIS7IFG6Z34LLJBR52KRKYRMEJA, seg_pos={0 0}, upload=507556]: Done uploading piece. err: <nil> Dec 29 18:28:13.203785268 (578h59m13.074868313s) [noise=true , address=77.90.31.11:4555 , piece_id=MUIQQLAPN3BS2J6TCVCHR3HWWMJGEJ2NUY4DAANCIAJIBWGZJIQA, seg_pos={0 0}, upload=507556]: Done uploading piece. err: <nil> Dec 29 18:28:13.222090063 (578h59m13.093173098s) [noise=true , address=45.91.116.12:4228 , piece_id=RVAXIMCSJBQNVMZFR4ZJS3OHV23V5S4RE6IACHR47YELFCJJXC2A, seg_pos={0 0}, upload=507556]: Done uploading piece. err: <nil> Dec 29 18:28:13.256296186 (578h59m13.127379221s) [noise=true , address=145.79.227.52:4446 , piece_id=A7UCPYY2MW5YC3YL3QMUXWVFKLVT7HNCO4L6YJVXI2I2L5W7KEVA, seg_pos={0 0}, upload=507556]: Done uploading piece. err: <nil> Dec 29 18:28:13.256314422 (578h59m13.127397457s) [seg_pos={0 0}, upload=507556]: Segment reached optimal threshold of 13 pieces. Dec 29 18:28:13.256489224 (578h59m13.127572259s) [noise=true , address=145.79.227.43:4159 , piece_id=6WYIZL7I3OT22XWKEIFPXLXEPQXR3BP3W4LPOEB4D6WZ7WHTG66A, seg_pos={0 0}, upload=507556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 18:28:13.256495266 (578h59m13.127578311s) [noise=true , address=145.79.227.43:4159 , piece_id=6WYIZL7I3OT22XWKEIFPXLXEPQXR3BP3W4LPOEB4D6WZ7WHTG66A, seg_pos={0 0}, upload=507556]: Long tail cancellation due to optimal threshold Dec 29 18:28:13.256582662 (578h59m13.127665697s) [noise=true , address=145.79.227.41:4086 , piece_id=6LV6JZJ6XJQHGUJXJTKE5C6LYOJSL6AZWHBCVOKFYOHUGHO5PSUA, seg_pos={0 0}, upload=507556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:57072->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:57072->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57072->145.79.227.41:4086: use of closed network connection Dec 29 18:28:13.256594996 (578h59m13.127678041s) [noise=true , address=145.79.227.41:4086 , piece_id=6LV6JZJ6XJQHGUJXJTKE5C6LYOJSL6AZWHBCVOKFYOHUGHO5PSUA, seg_pos={0 0}, upload=507556]: Long tail cancellation due to optimal threshold Dec 29 18:28:13.256673505 (578h59m13.12775654s) [noise=true , address=45.91.116.5:4300 , piece_id=GNZTFH554AU37PQG7MS3JZ3W5ZZD6M5APBFRTTY3YYAIG2GSTLXA, seg_pos={0 0}, upload=507556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 29 18:28:13.256680588 (578h59m13.127763623s) [noise=true , address=45.91.116.5:4300 , piece_id=GNZTFH554AU37PQG7MS3JZ3W5ZZD6M5APBFRTTY3YYAIG2GSTLXA, seg_pos={0 0}, upload=507556]: Long tail cancellation due to optimal threshold Dec 29 18:28:13.256698132 (578h59m13.127781167s) [seg_pos={0 0}, upload=507556]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:28:13.256760511 (578h59m13.127843546s) [upload=507556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:28:13.256765811 (578h59m13.127848846s) [upload=507556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:28:13.547804253 (578h59m13.418887288s) [upload=507556]: Done uploading segments... Dec 29 18:28:13.547823409 (578h59m13.418906454s) [upload=507556]: Upload finished. err: <nil> Duration: 30.003853111s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 40.833592318s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:28:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 40.833621964s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.530066377s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:27:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.530094951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 15.383994901s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:27:29 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 15.384022213s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X......XXX.] duration: 17.564178354s trace: https://jaeger.dev.storj.io/trace/3a1fd42ac932b285 FAIL (2025-12-29 18:27:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: R4OWZJTOAVKLUSSKN6PVH64LBZYD5AECMX6KNQEOX42CZ25LPP6A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: OYHRDD6CBS3HHHY5Q2QOAKUXD7SDEZHON5WIVS6IIIDQLEMM3YTQ) context deadline exceeded ======LOGS====== 18:26:33.813705818 (578h57m33.684788863s) [upload=507532]: Starting upload Dec 29 18:26:33.813874650 (578h57m33.684957705s) [upload=507532]: Done starting upload Dec 29 18:26:33.813958349 (578h57m33.685041394s) [upload=507532]: Uploading segments... Dec 29 18:26:33.814412383 (578h57m33.685495418s) [upload=507532]: Got next segment. Inline: false Dec 29 18:26:33.814457921 (578h57m33.685540966s) [upload=507532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:26:33.814471216 (578h57m33.685554261s) [upload=507532]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:26:34.063810996 (578h57m33.934894041s) [seg_pos={0 0}, upload=507532]: Begin upload segment... Dec 29 18:26:34.063864398 (578h57m33.934947433s) [seg_pos={0 0}, upload=507532]: Done begin upload segment. Dec 29 18:26:34.063896880 (578h57m33.934979925s) [noise=true , address=77.90.31.12:4590 , piece_id=R4OWZJTOAVKLUSSKN6PVH64LBZYD5AECMX6KNQEOX42CZ25LPP6A, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.063974087 (578h57m33.935057122s) [noise=true , address=145.79.227.39:4015 , piece_id=VHNJ57KTDBQDMNWOCTSNITXNRDYGWPSG4U47UKDZ2RDT77LXO7OA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.064553570 (578h57m33.935636606s) [noise=true , address=145.79.227.41:4086 , piece_id=7UZSYMYEHFEZGULD5L7O5UPXWQZQ3P6D53WAFNEAJYU6SCDMKX3A, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.064878950 (578h57m33.935961995s) [noise=true , address=145.79.227.26:4655 , piece_id=JRLU3VWUGKGAAUT3WVXWH75BUYJKMXMZ7MCKSWKANV5ZTEQV7FEA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.065159133 (578h57m33.936242168s) [noise=true , address=145.79.227.24:4581 , piece_id=NAM6O6TWOGWTHSLW4T62I2KZHBZVXN3HDURV7CNJI3MHFV6G3XJA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.065533095 (578h57m33.936616131s) [noise=true , address=145.79.227.11:4729 , piece_id=PZ52MD27J7SPXJQLCW7RJ4P2XECZZOTFTJVP64TX543M6FB2CYBA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.066004624 (578h57m33.937087659s) [noise=true , address=145.79.227.32:4835 , piece_id=ODK3ZQEKNL5Z4TBZXC2YZMME4NGGBPQRDO2U2V7ZF5EKG56F5PNA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.066309954 (578h57m33.93739299s) [noise=true , address=145.79.227.2:4115 , piece_id=J3VBXGNF4YC4EZYOJCKTKCDSWTB6PTUQMN5NDPB26HXALTR5SRDQ, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.066589918 (578h57m33.937672953s) [noise=true , address=145.79.227.8:4329 , piece_id=PJAES7N5AJF5ZYOLBI5WFKKNPHBGK5NL4KJF7EMDMK4QMQTQJDTQ, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.068018888 (578h57m33.939101924s) [noise=true , address=145.79.227.15:4874 , piece_id=JLM2YQDE5Y337HVT7V2RO7CXX6EE74TTRSS5SPOIPVBAQVYIKWAA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.068812069 (578h57m33.939895114s) [noise=true , address=45.91.116.7:4010 , piece_id=5EAHIRLEJHGPFI5M3VBGE33H437YCYJQ4Y47UQU7OCJ3KOVJ4FDA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.070383621 (578h57m33.941466656s) [noise=true , address=145.79.227.51:4411 , piece_id=4IKHN4M7Q65OOU4MUTU5XJBNZC2DVCBEOOPA4S4KGJW2DOPSFXDA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.071867497 (578h57m33.942950532s) [noise=true , address=145.79.227.52:4445 , piece_id=JE6VULINM2UHOSPP5POHIPYCKNXK7HO2U3XEVCGVH3ZNYPTMCB7A, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.073144719 (578h57m33.944227754s) [noise=true , address=145.79.227.4:4185 , piece_id=XDMVQTKV5JCIAVZMC4PT2YQPM5EXO4YO3N75QHUGAAHUJK76RH5Q, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.074638453 (578h57m33.945721488s) [noise=true , address=45.91.116.5:4300 , piece_id=OYHRDD6CBS3HHHY5Q2QOAKUXD7SDEZHON5WIVS6IIIDQLEMM3YTQ, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.076525787 (578h57m33.947608832s) [noise=true , address=77.90.31.14:4662 , piece_id=4DWEBBGFSBIVG4MTIM6HGLPPVJMRTODFPWTVSIO7HJZJTFXAT2AA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:35.761702838 (578h57m35.632785873s) [upload=507532]: Next returned nil segment Dec 29 18:26:35.761724058 (578h57m35.632807113s) [upload=507532]: Waiting for error group managing segments... Dec 29 18:26:36.033563302 (578h57m35.904646347s) [noise=true , address=145.79.227.11:4729 , piece_id=PZ52MD27J7SPXJQLCW7RJ4P2XECZZOTFTJVP64TX543M6FB2CYBA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.052393928 (578h57m35.923476963s) [noise=true , address=145.79.227.24:4581 , piece_id=NAM6O6TWOGWTHSLW4T62I2KZHBZVXN3HDURV7CNJI3MHFV6G3XJA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.057903458 (578h57m35.928986503s) [noise=true , address=145.79.227.4:4185 , piece_id=XDMVQTKV5JCIAVZMC4PT2YQPM5EXO4YO3N75QHUGAAHUJK76RH5Q, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.060886178 (578h57m35.931969213s) [noise=true , address=145.79.227.2:4115 , piece_id=J3VBXGNF4YC4EZYOJCKTKCDSWTB6PTUQMN5NDPB26HXALTR5SRDQ, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.068023248 (578h57m35.939106294s) [noise=true , address=145.79.227.8:4329 , piece_id=PJAES7N5AJF5ZYOLBI5WFKKNPHBGK5NL4KJF7EMDMK4QMQTQJDTQ, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.069710270 (578h57m35.940793315s) [noise=true , address=145.79.227.32:4835 , piece_id=ODK3ZQEKNL5Z4TBZXC2YZMME4NGGBPQRDO2U2V7ZF5EKG56F5PNA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.075557525 (578h57m35.94664056s) [noise=true , address=145.79.227.15:4874 , piece_id=JLM2YQDE5Y337HVT7V2RO7CXX6EE74TTRSS5SPOIPVBAQVYIKWAA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.086714258 (578h57m35.957797294s) [noise=true , address=145.79.227.26:4655 , piece_id=JRLU3VWUGKGAAUT3WVXWH75BUYJKMXMZ7MCKSWKANV5ZTEQV7FEA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.087295215 (578h57m35.95837825s) [noise=true , address=145.79.227.39:4015 , piece_id=VHNJ57KTDBQDMNWOCTSNITXNRDYGWPSG4U47UKDZ2RDT77LXO7OA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.094639470 (578h57m35.965722505s) [noise=true , address=145.79.227.52:4445 , piece_id=JE6VULINM2UHOSPP5POHIPYCKNXK7HO2U3XEVCGVH3ZNYPTMCB7A, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.233550958 (578h57m36.104633993s) [noise=true , address=77.90.31.14:4662 , piece_id=4DWEBBGFSBIVG4MTIM6HGLPPVJMRTODFPWTVSIO7HJZJTFXAT2AA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.256068244 (578h57m36.127151279s) [noise=true , address=77.90.31.12:4590 , piece_id=R4OWZJTOAVKLUSSKN6PVH64LBZYD5AECMX6KNQEOX42CZ25LPP6A, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.322162147 (578h57m36.193245182s) [noise=true , address=45.91.116.5:4300 , piece_id=OYHRDD6CBS3HHHY5Q2QOAKUXD7SDEZHON5WIVS6IIIDQLEMM3YTQ, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.322185030 (578h57m36.193268065s) [seg_pos={0 0}, upload=507532]: Segment reached optimal threshold of 13 pieces. Dec 29 18:26:36.322387746 (578h57m36.193470791s) [noise=true , address=45.91.116.7:4010 , piece_id=5EAHIRLEJHGPFI5M3VBGE33H437YCYJQ4Y47UQU7OCJ3KOVJ4FDA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 18:26:36.322396553 (578h57m36.193479588s) [noise=true , address=45.91.116.7:4010 , piece_id=5EAHIRLEJHGPFI5M3VBGE33H437YCYJQ4Y47UQU7OCJ3KOVJ4FDA, seg_pos={0 0}, upload=507532]: Long tail cancellation due to optimal threshold Dec 29 18:26:36.322401503 (578h57m36.193484538s) [noise=true , address=145.79.227.41:4086 , piece_id=7UZSYMYEHFEZGULD5L7O5UPXWQZQ3P6D53WAFNEAJYU6SCDMKX3A, seg_pos={0 0}, upload=507532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:59776->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:59776->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59776->145.79.227.41:4086: use of closed network connection Dec 29 18:26:36.322410490 (578h57m36.193493525s) [noise=true , address=145.79.227.41:4086 , piece_id=7UZSYMYEHFEZGULD5L7O5UPXWQZQ3P6D53WAFNEAJYU6SCDMKX3A, seg_pos={0 0}, upload=507532]: Long tail cancellation due to optimal threshold Dec 29 18:26:36.322545437 (578h57m36.193628472s) [noise=true , address=145.79.227.51:4411 , piece_id=4IKHN4M7Q65OOU4MUTU5XJBNZC2DVCBEOOPA4S4KGJW2DOPSFXDA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: expected piece hash; context canceled; EOF Dec 29 18:26:36.322553001 (578h57m36.193636046s) [noise=true , address=145.79.227.51:4411 , piece_id=4IKHN4M7Q65OOU4MUTU5XJBNZC2DVCBEOOPA4S4KGJW2DOPSFXDA, seg_pos={0 0}, upload=507532]: Long tail cancellation due to optimal threshold Dec 29 18:26:36.322574341 (578h57m36.193657386s) [seg_pos={0 0}, upload=507532]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:26:36.322812254 (578h57m36.19389529s) [upload=507532]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:26:36.322820801 (578h57m36.193903847s) [upload=507532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:26:36.616226831 (578h57m36.487309866s) [upload=507532]: Done uploading segments... Dec 29 18:26:36.616245626 (578h57m36.487328661s) [upload=507532]: Upload finished. err: <nil> Duration: 17.564264097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X......XXX] duration: 30.002428129s trace: https://jaeger.dev.storj.io/trace/3a1fd42ac932b285 FAIL (2025-12-29 18:27:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: R4OWZJTOAVKLUSSKN6PVH64LBZYD5AECMX6KNQEOX42CZ25LPP6A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: OYHRDD6CBS3HHHY5Q2QOAKUXD7SDEZHON5WIVS6IIIDQLEMM3YTQ) context deadline exceeded ======LOGS====== 18:26:33.813705818 (578h57m33.684788863s) [upload=507532]: Starting upload Dec 29 18:26:33.813874650 (578h57m33.684957705s) [upload=507532]: Done starting upload Dec 29 18:26:33.813958349 (578h57m33.685041394s) [upload=507532]: Uploading segments... Dec 29 18:26:33.814412383 (578h57m33.685495418s) [upload=507532]: Got next segment. Inline: false Dec 29 18:26:33.814457921 (578h57m33.685540966s) [upload=507532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:26:33.814471216 (578h57m33.685554261s) [upload=507532]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:26:34.063810996 (578h57m33.934894041s) [seg_pos={0 0}, upload=507532]: Begin upload segment... Dec 29 18:26:34.063864398 (578h57m33.934947433s) [seg_pos={0 0}, upload=507532]: Done begin upload segment. Dec 29 18:26:34.063896880 (578h57m33.934979925s) [noise=true , address=77.90.31.12:4590 , piece_id=R4OWZJTOAVKLUSSKN6PVH64LBZYD5AECMX6KNQEOX42CZ25LPP6A, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.063974087 (578h57m33.935057122s) [noise=true , address=145.79.227.39:4015 , piece_id=VHNJ57KTDBQDMNWOCTSNITXNRDYGWPSG4U47UKDZ2RDT77LXO7OA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.064553570 (578h57m33.935636606s) [noise=true , address=145.79.227.41:4086 , piece_id=7UZSYMYEHFEZGULD5L7O5UPXWQZQ3P6D53WAFNEAJYU6SCDMKX3A, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.064878950 (578h57m33.935961995s) [noise=true , address=145.79.227.26:4655 , piece_id=JRLU3VWUGKGAAUT3WVXWH75BUYJKMXMZ7MCKSWKANV5ZTEQV7FEA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.065159133 (578h57m33.936242168s) [noise=true , address=145.79.227.24:4581 , piece_id=NAM6O6TWOGWTHSLW4T62I2KZHBZVXN3HDURV7CNJI3MHFV6G3XJA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.065533095 (578h57m33.936616131s) [noise=true , address=145.79.227.11:4729 , piece_id=PZ52MD27J7SPXJQLCW7RJ4P2XECZZOTFTJVP64TX543M6FB2CYBA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.066004624 (578h57m33.937087659s) [noise=true , address=145.79.227.32:4835 , piece_id=ODK3ZQEKNL5Z4TBZXC2YZMME4NGGBPQRDO2U2V7ZF5EKG56F5PNA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.066309954 (578h57m33.93739299s) [noise=true , address=145.79.227.2:4115 , piece_id=J3VBXGNF4YC4EZYOJCKTKCDSWTB6PTUQMN5NDPB26HXALTR5SRDQ, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.066589918 (578h57m33.937672953s) [noise=true , address=145.79.227.8:4329 , piece_id=PJAES7N5AJF5ZYOLBI5WFKKNPHBGK5NL4KJF7EMDMK4QMQTQJDTQ, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.068018888 (578h57m33.939101924s) [noise=true , address=145.79.227.15:4874 , piece_id=JLM2YQDE5Y337HVT7V2RO7CXX6EE74TTRSS5SPOIPVBAQVYIKWAA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.068812069 (578h57m33.939895114s) [noise=true , address=45.91.116.7:4010 , piece_id=5EAHIRLEJHGPFI5M3VBGE33H437YCYJQ4Y47UQU7OCJ3KOVJ4FDA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.070383621 (578h57m33.941466656s) [noise=true , address=145.79.227.51:4411 , piece_id=4IKHN4M7Q65OOU4MUTU5XJBNZC2DVCBEOOPA4S4KGJW2DOPSFXDA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.071867497 (578h57m33.942950532s) [noise=true , address=145.79.227.52:4445 , piece_id=JE6VULINM2UHOSPP5POHIPYCKNXK7HO2U3XEVCGVH3ZNYPTMCB7A, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.073144719 (578h57m33.944227754s) [noise=true , address=145.79.227.4:4185 , piece_id=XDMVQTKV5JCIAVZMC4PT2YQPM5EXO4YO3N75QHUGAAHUJK76RH5Q, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.074638453 (578h57m33.945721488s) [noise=true , address=45.91.116.5:4300 , piece_id=OYHRDD6CBS3HHHY5Q2QOAKUXD7SDEZHON5WIVS6IIIDQLEMM3YTQ, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.076525787 (578h57m33.947608832s) [noise=true , address=77.90.31.14:4662 , piece_id=4DWEBBGFSBIVG4MTIM6HGLPPVJMRTODFPWTVSIO7HJZJTFXAT2AA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:35.761702838 (578h57m35.632785873s) [upload=507532]: Next returned nil segment Dec 29 18:26:35.761724058 (578h57m35.632807113s) [upload=507532]: Waiting for error group managing segments... Dec 29 18:26:36.033563302 (578h57m35.904646347s) [noise=true , address=145.79.227.11:4729 , piece_id=PZ52MD27J7SPXJQLCW7RJ4P2XECZZOTFTJVP64TX543M6FB2CYBA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.052393928 (578h57m35.923476963s) [noise=true , address=145.79.227.24:4581 , piece_id=NAM6O6TWOGWTHSLW4T62I2KZHBZVXN3HDURV7CNJI3MHFV6G3XJA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.057903458 (578h57m35.928986503s) [noise=true , address=145.79.227.4:4185 , piece_id=XDMVQTKV5JCIAVZMC4PT2YQPM5EXO4YO3N75QHUGAAHUJK76RH5Q, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.060886178 (578h57m35.931969213s) [noise=true , address=145.79.227.2:4115 , piece_id=J3VBXGNF4YC4EZYOJCKTKCDSWTB6PTUQMN5NDPB26HXALTR5SRDQ, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.068023248 (578h57m35.939106294s) [noise=true , address=145.79.227.8:4329 , piece_id=PJAES7N5AJF5ZYOLBI5WFKKNPHBGK5NL4KJF7EMDMK4QMQTQJDTQ, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.069710270 (578h57m35.940793315s) [noise=true , address=145.79.227.32:4835 , piece_id=ODK3ZQEKNL5Z4TBZXC2YZMME4NGGBPQRDO2U2V7ZF5EKG56F5PNA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.075557525 (578h57m35.94664056s) [noise=true , address=145.79.227.15:4874 , piece_id=JLM2YQDE5Y337HVT7V2RO7CXX6EE74TTRSS5SPOIPVBAQVYIKWAA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.086714258 (578h57m35.957797294s) [noise=true , address=145.79.227.26:4655 , piece_id=JRLU3VWUGKGAAUT3WVXWH75BUYJKMXMZ7MCKSWKANV5ZTEQV7FEA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.087295215 (578h57m35.95837825s) [noise=true , address=145.79.227.39:4015 , piece_id=VHNJ57KTDBQDMNWOCTSNITXNRDYGWPSG4U47UKDZ2RDT77LXO7OA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.094639470 (578h57m35.965722505s) [noise=true , address=145.79.227.52:4445 , piece_id=JE6VULINM2UHOSPP5POHIPYCKNXK7HO2U3XEVCGVH3ZNYPTMCB7A, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.233550958 (578h57m36.104633993s) [noise=true , address=77.90.31.14:4662 , piece_id=4DWEBBGFSBIVG4MTIM6HGLPPVJMRTODFPWTVSIO7HJZJTFXAT2AA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.256068244 (578h57m36.127151279s) [noise=true , address=77.90.31.12:4590 , piece_id=R4OWZJTOAVKLUSSKN6PVH64LBZYD5AECMX6KNQEOX42CZ25LPP6A, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.322162147 (578h57m36.193245182s) [noise=true , address=45.91.116.5:4300 , piece_id=OYHRDD6CBS3HHHY5Q2QOAKUXD7SDEZHON5WIVS6IIIDQLEMM3YTQ, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.322185030 (578h57m36.193268065s) [seg_pos={0 0}, upload=507532]: Segment reached optimal threshold of 13 pieces. Dec 29 18:26:36.322387746 (578h57m36.193470791s) [noise=true , address=45.91.116.7:4010 , piece_id=5EAHIRLEJHGPFI5M3VBGE33H437YCYJQ4Y47UQU7OCJ3KOVJ4FDA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 18:26:36.322396553 (578h57m36.193479588s) [noise=true , address=45.91.116.7:4010 , piece_id=5EAHIRLEJHGPFI5M3VBGE33H437YCYJQ4Y47UQU7OCJ3KOVJ4FDA, seg_pos={0 0}, upload=507532]: Long tail cancellation due to optimal threshold Dec 29 18:26:36.322401503 (578h57m36.193484538s) [noise=true , address=145.79.227.41:4086 , piece_id=7UZSYMYEHFEZGULD5L7O5UPXWQZQ3P6D53WAFNEAJYU6SCDMKX3A, seg_pos={0 0}, upload=507532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:59776->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:59776->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59776->145.79.227.41:4086: use of closed network connection Dec 29 18:26:36.322410490 (578h57m36.193493525s) [noise=true , address=145.79.227.41:4086 , piece_id=7UZSYMYEHFEZGULD5L7O5UPXWQZQ3P6D53WAFNEAJYU6SCDMKX3A, seg_pos={0 0}, upload=507532]: Long tail cancellation due to optimal threshold Dec 29 18:26:36.322545437 (578h57m36.193628472s) [noise=true , address=145.79.227.51:4411 , piece_id=4IKHN4M7Q65OOU4MUTU5XJBNZC2DVCBEOOPA4S4KGJW2DOPSFXDA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: expected piece hash; context canceled; EOF Dec 29 18:26:36.322553001 (578h57m36.193636046s) [noise=true , address=145.79.227.51:4411 , piece_id=4IKHN4M7Q65OOU4MUTU5XJBNZC2DVCBEOOPA4S4KGJW2DOPSFXDA, seg_pos={0 0}, upload=507532]: Long tail cancellation due to optimal threshold Dec 29 18:26:36.322574341 (578h57m36.193657386s) [seg_pos={0 0}, upload=507532]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:26:36.322812254 (578h57m36.19389529s) [upload=507532]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:26:36.322820801 (578h57m36.193903847s) [upload=507532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:26:36.616226831 (578h57m36.487309866s) [upload=507532]: Done uploading segments... Dec 29 18:26:36.616245626 (578h57m36.487328661s) [upload=507532]: Upload finished. err: <nil> Duration: 30.00244491s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.460620759s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:26:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.460644524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 35.539971698s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:26:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 35.539997275s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.382566376s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:26:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.382596544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 15.462960387s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:25:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 15.462997428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X......XX] duration: 30.00361769s trace: https://jaeger.dev.storj.io/trace/322cb0b33f6f935 FAIL (2025-12-29 18:25:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: J6IVG5NQ4JYSZ7ZB6APN2JVEF74MMLGEFOPUI34FPS74HH77ALUA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: RJLQMV3FZEVPFR4EORZP7JOGC5M7YARZVY2RVRLWLQELZPCGZKGQ) context deadline exceeded ======LOGS====== Dec 29 18:25:27.495891612 (578h56m27.366974647s) [upload=507515]: Starting upload Dec 29 18:25:27.495951246 (578h56m27.367034281s) [upload=507515]: Done starting upload Dec 29 18:25:27.496025938 (578h56m27.367108973s) [upload=507515]: Uploading segments... Dec 29 18:25:27.496192846 (578h56m27.367275881s) [upload=507515]: Got next segment. Inline: false Dec 29 18:25:27.496235757 (578h56m27.367318792s) [upload=507515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:25:27.496240937 (578h56m27.367323982s) [upload=507515]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:25:27.744036697 (578h56m27.615119742s) [seg_pos={0 0}, upload=507515]: Begin upload segment... Dec 29 18:25:27.744077726 (578h56m27.615160771s) [seg_pos={0 0}, upload=507515]: Done begin upload segment. Dec 29 18:25:27.744100990 (578h56m27.615184035s) [noise=true , address=77.90.31.15:4700 , piece_id=FSKTQLLC5UX3DHT3FVYR7NFETUYIF4YGZOM2ROD6YSKCGUD7RNGA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.744662880 (578h56m27.615745915s) [noise=true , address=145.79.227.23:4546 , piece_id=CLG6DGNNXSAEQQRGIZP22OPJINJ3POMGAT6YMDAWNKAMF5FANU5Q, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.744963633 (578h56m27.616046678s) [noise=true , address=45.91.116.13:4154 , piece_id=Z5VJ6TRVXCTHPWGDXUTXMESZQYTHA4AUAFW43JCH3N4UIUG7KTKA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.745406917 (578h56m27.616489952s) [noise=true , address=77.90.31.19:4842 , piece_id=BLSNHSS25ASTUSCQOJNZI2IE7FBRYU6COPT5NXNUPAWM3VYPWZPA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.745716988 (578h56m27.616800033s) [noise=true , address=77.90.31.10:4517 , piece_id=J6IVG5NQ4JYSZ7ZB6APN2JVEF74MMLGEFOPUI34FPS74HH77ALUA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.746137188 (578h56m27.617220223s) [noise=true , address=45.91.116.5:4300 , piece_id=S62LTREU5MH24S455F3XXS3RDIA4M6ESWNXFVJFZDHB5CJBVNIYQ, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.746466194 (578h56m27.617549229s) [noise=true , address=145.79.227.44:4194 , piece_id=TC3QIVFM6SGWEGCZ3Q4OFT2C5TBXMH6KOPXGCYJO6MDZHZJUPBDQ, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.746815499 (578h56m27.617898534s) [noise=true , address=45.91.116.12:4225 , piece_id=QA6HDQPPKCG5VB5CCKNUKHIZIVCH6EEYNHIGTSU4M4LODB5IMZCA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.747198949 (578h56m27.618281984s) [noise=true , address=45.91.116.9:4261 , piece_id=RJLQMV3FZEVPFR4EORZP7JOGC5M7YARZVY2RVRLWLQELZPCGZKGQ, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.747603971 (578h56m27.618687016s) [noise=true , address=145.79.227.36:4518 , piece_id=GWZESEOH2M3RHLP7YIZADMSCR2AJT34IE5UEAUPJMRNEOPL2G7WQ, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.747961211 (578h56m27.619044256s) [noise=true , address=45.91.116.4:4940 , piece_id=2LTKTTOISUK67GRVB34XZ2CYKJANG4YFE6G3JQMYPCMEKMLIKUEQ, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.748363567 (578h56m27.619446612s) [noise=true , address=145.79.227.4:4187 , piece_id=WCDZ2XYU3HWL2R67PYERAO3XL72UFGCIHLL3CNIXY5AVM3KEWGZA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.749220548 (578h56m27.620303594s) [noise=true , address=145.79.227.22:4509 , piece_id=D3VU3RLU4IDKJKAO7NYDD4EFL4MJ42TKAI3SDJIAQI5Y4ROSDMSQ, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.749499299 (578h56m27.620582344s) [noise=true , address=145.79.227.41:4085 , piece_id=LXRNUVZTQ7O3YXYFKXKWCJNHGWPYRF76FCDFTL4LHALOQ4AZBAJA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.749842964 (578h56m27.620926009s) [noise=true , address=145.79.227.17:4972 , piece_id=4C2NLVFV2V6RSL3NPJ4BAFZ2JYHWDTMACNMLJFS6TENYDH7LALDA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.750152183 (578h56m27.621235228s) [noise=true , address=145.79.227.51:4411 , piece_id=DHTKNBIGFPA4SCDME37BUAOLZCRLY7B35FKGDWVHJQ6WDFKB44PA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:29.353321468 (578h56m29.224404513s) [upload=507515]: Next returned nil segment Dec 29 18:25:29.353335826 (578h56m29.224418871s) [upload=507515]: Waiting for error group managing segments... Dec 29 18:25:29.725986288 (578h56m29.597069323s) [noise=true , address=145.79.227.4:4187 , piece_id=WCDZ2XYU3HWL2R67PYERAO3XL72UFGCIHLL3CNIXY5AVM3KEWGZA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:29.735253815 (578h56m29.60633685s) [noise=true , address=145.79.227.17:4972 , piece_id=4C2NLVFV2V6RSL3NPJ4BAFZ2JYHWDTMACNMLJFS6TENYDH7LALDA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:29.735668664 (578h56m29.606751699s) [noise=true , address=145.79.227.36:4518 , piece_id=GWZESEOH2M3RHLP7YIZADMSCR2AJT34IE5UEAUPJMRNEOPL2G7WQ, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:29.736204014 (578h56m29.60728705s) [noise=true , address=145.79.227.51:4411 , piece_id=DHTKNBIGFPA4SCDME37BUAOLZCRLY7B35FKGDWVHJQ6WDFKB44PA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:29.751744596 (578h56m29.622827641s) [noise=true , address=145.79.227.23:4546 , piece_id=CLG6DGNNXSAEQQRGIZP22OPJINJ3POMGAT6YMDAWNKAMF5FANU5Q, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:29.763374571 (578h56m29.634457616s) [noise=true , address=145.79.227.22:4509 , piece_id=D3VU3RLU4IDKJKAO7NYDD4EFL4MJ42TKAI3SDJIAQI5Y4ROSDMSQ, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:29.767005565 (578h56m29.63808861s) [noise=true , address=145.79.227.41:4085 , piece_id=LXRNUVZTQ7O3YXYFKXKWCJNHGWPYRF76FCDFTL4LHALOQ4AZBAJA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:29.920881859 (578h56m29.791964894s) [noise=true , address=77.90.31.10:4517 , piece_id=J6IVG5NQ4JYSZ7ZB6APN2JVEF74MMLGEFOPUI34FPS74HH77ALUA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:29.971256640 (578h56m29.842339675s) [noise=true , address=45.91.116.13:4154 , piece_id=Z5VJ6TRVXCTHPWGDXUTXMESZQYTHA4AUAFW43JCH3N4UIUG7KTKA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:30.096541321 (578h56m29.967624366s) [noise=true , address=45.91.116.5:4300 , piece_id=S62LTREU5MH24S455F3XXS3RDIA4M6ESWNXFVJFZDHB5CJBVNIYQ, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:30.103949577 (578h56m29.975032622s) [noise=true , address=77.90.31.15:4700 , piece_id=FSKTQLLC5UX3DHT3FVYR7NFETUYIF4YGZOM2ROD6YSKCGUD7RNGA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:30.104446393 (578h56m29.975529438s) [noise=true , address=45.91.116.9:4261 , piece_id=RJLQMV3FZEVPFR4EORZP7JOGC5M7YARZVY2RVRLWLQELZPCGZKGQ, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:30.751295472 (578h56m30.622378517s) [noise=true , address=45.91.116.12:4225 , piece_id=QA6HDQPPKCG5VB5CCKNUKHIZIVCH6EEYNHIGTSU4M4LODB5IMZCA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:30.751320850 (578h56m30.622403895s) [seg_pos={0 0}, upload=507515]: Segment reached optimal threshold of 13 pieces. Dec 29 18:25:30.751538815 (578h56m30.62262186s) [noise=true , address=77.90.31.19:4842 , piece_id=BLSNHSS25ASTUSCQOJNZI2IE7FBRYU6COPT5NXNUPAWM3VYPWZPA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 29 18:25:30.751549245 (578h56m30.62263229s) [noise=true , address=77.90.31.19:4842 , piece_id=BLSNHSS25ASTUSCQOJNZI2IE7FBRYU6COPT5NXNUPAWM3VYPWZPA, seg_pos={0 0}, upload=507515]: Long tail cancellation due to optimal threshold Dec 29 18:25:30.751736482 (578h56m30.622819517s) [noise=true , address=145.79.227.44:4194 , piece_id=TC3QIVFM6SGWEGCZ3Q4OFT2C5TBXMH6KOPXGCYJO6MDZHZJUPBDQ, seg_pos={0 0}, upload=507515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:25:30.751758403 (578h56m30.622841458s) [noise=true , address=145.79.227.44:4194 , piece_id=TC3QIVFM6SGWEGCZ3Q4OFT2C5TBXMH6KOPXGCYJO6MDZHZJUPBDQ, seg_pos={0 0}, upload=507515]: Long tail cancellation due to optimal threshold Dec 29 18:25:30.751765436 (578h56m30.622848471s) [noise=true , address=45.91.116.4:4940 , piece_id=2LTKTTOISUK67GRVB34XZ2CYKJANG4YFE6G3JQMYPCMEKMLIKUEQ, seg_pos={0 0}, upload=507515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 29 18:25:30.751778552 (578h56m30.622861597s) [noise=true , address=45.91.116.4:4940 , piece_id=2LTKTTOISUK67GRVB34XZ2CYKJANG4YFE6G3JQMYPCMEKMLIKUEQ, seg_pos={0 0}, upload=507515]: Long tail cancellation due to optimal threshold Dec 29 18:25:30.751799841 (578h56m30.622882877s) [seg_pos={0 0}, upload=507515]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:25:30.751885245 (578h56m30.62296829s) [upload=507515]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:25:30.751892158 (578h56m30.622975193s) [upload=507515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:25:31.050056256 (578h56m30.921139291s) [upload=507515]: Done uploading segments... Dec 29 18:25:31.050082466 (578h56m30.921165501s) [upload=507515]: Upload finished. err: <nil> Duration: 30.003643449s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.334530008s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:25:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.334552379s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.387528827s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:25:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.387555086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 30.579649143s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:25:18 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 30.579678671s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X......X] duration: 30.003197506s trace: https://jaeger.dev.storj.io/trace/5bcb545882f80b44 FAIL (2025-12-29 18:24:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: QL2VPDIXIYTBKJ3LHTMXUQG47ANBBINESBZSUYJRCFS4N5HTBLHQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: GWQK2PWNQQ3EOPGXNY3HITL526EPKZI2E6RKXV4F3OFQNT4VX6OQ) context deadline exceeded ======LOGS====== gments... Dec 29 18:24:13.483356826 (578h55m13.354439871s) [upload=507499]: Got next segment. Inline: false Dec 29 18:24:13.483401041 (578h55m13.354484076s) [upload=507499]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:24:13.483406441 (578h55m13.354489486s) [upload=507499]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:24:13.748310639 (578h55m13.619393684s) [seg_pos={0 0}, upload=507499]: Begin upload segment... Dec 29 18:24:13.748364862 (578h55m13.619447897s) [seg_pos={0 0}, upload=507499]: Done begin upload segment. Dec 29 18:24:13.748406632 (578h55m13.619489707s) [noise=true , address=45.91.116.8:4045 , piece_id=OMCXDTMQNC7CVXRXBHCS3NXHAEG5WHHDSKSF5FPJOGM2AWUZWHCA, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.748507063 (578h55m13.619590108s) [noise=true , address=145.79.227.32:4836 , piece_id=DVD6Y54RSRSQT656SWMK32HA5MTJKYQANXVQLLQTD7UNEDECCRHQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.749078200 (578h55m13.620161245s) [noise=true , address=145.79.227.16:4909 , piece_id=XKGMGNTEIBSWHNHAUKAHR7D3DDOTO7QTBHE5KDHWH6UKNWIC6PBA, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.749417587 (578h55m13.620500632s) [noise=true , address=45.91.116.7:4012 , piece_id=ZKE374IL4M5LW2KC7JNLZK7DFPBQPOYQMIPXALC2DJ3R7RUWBOFQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.749878193 (578h55m13.620961239s) [noise=true , address=45.91.116.9:4264 , piece_id=QL2VPDIXIYTBKJ3LHTMXUQG47ANBBINESBZSUYJRCFS4N5HTBLHQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.750233109 (578h55m13.621316144s) [noise=true , address=145.79.227.11:4729 , piece_id=OJ2OYREMNIIWOOB63ZSSKUXI5L6GPJQBE7LFSGE766PFOYJQWVQA, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.750596111 (578h55m13.621679146s) [noise=true , address=145.79.227.50:4375 , piece_id=NA35IP4WGF5VPUPQFOKKPCK3BH5EKVEDP5WCHIHOC6OYISVNGAWQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.750958841 (578h55m13.622041886s) [noise=true , address=145.79.227.2:4113 , piece_id=KLRRZ6N6TIFNRMUO7QTAO6IEI7WOA2XWICPUQWXGV4QJKRHTR67Q, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.751381486 (578h55m13.622464521s) [noise=true , address=45.91.116.14:4190 , piece_id=GWQK2PWNQQ3EOPGXNY3HITL526EPKZI2E6RKXV4F3OFQNT4VX6OQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.752396238 (578h55m13.623479313s) [noise=true , address=45.91.116.6:4974 , piece_id=F7GY2RNNW25WV7NK2M65VOJXUAGXFYZ62QB2OXZNEN3ANLKVY5MQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.753649785 (578h55m13.62473282s) [noise=true , address=145.79.227.3:4150 , piece_id=J7TG5GXUFRQC3MNFZEWUYDLF3N3DWIQR4CYYAEYBECADW6CWNOGA, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.754844190 (578h55m13.625927235s) [noise=true , address=145.79.227.6:4259 , piece_id=A73B6FI3W4257FJZ7PDYKCVOAK7FWXYOY5LAJO4MFF6WBBQEMHKQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.756837696 (578h55m13.627920731s) [noise=true , address=77.90.31.11:4556 , piece_id=6VH5X3XLYBXCA4VPWYY33Z34XJ3WEDLHVLPX4B7XYFALFN6HPKOQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.758451268 (578h55m13.629534303s) [noise=true , address=145.79.227.47:4266 , piece_id=QUMQLLHPAOBXGVT3EHGUO2PBHA3MVWSKQ6S3IISHUNECC7NU3OMA, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.759642376 (578h55m13.630725411s) [noise=true , address=145.79.227.36:4518 , piece_id=LJJIQM7QENYCHZXM7H2HE3TUNCXUWM453EZNH5HRMHXLGTDGQ4CA, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.760881596 (578h55m13.631964641s) [noise=true , address=77.90.31.14:4661 , piece_id=5HFUCZWSDRGXXKY2XF3J4TOK4P6VKPTZ56EIIR6CTADBGVUNDXEQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:15.448480408 (578h55m15.319563443s) [upload=507499]: Next returned nil segment Dec 29 18:24:15.448494124 (578h55m15.319577169s) [upload=507499]: Waiting for error group managing segments... Dec 29 18:24:15.732661511 (578h55m15.603744556s) [noise=true , address=145.79.227.32:4836 , piece_id=DVD6Y54RSRSQT656SWMK32HA5MTJKYQANXVQLLQTD7UNEDECCRHQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:15.741589742 (578h55m15.612672777s) [noise=true , address=145.79.227.3:4150 , piece_id=J7TG5GXUFRQC3MNFZEWUYDLF3N3DWIQR4CYYAEYBECADW6CWNOGA, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:15.748295451 (578h55m15.619378496s) [noise=true , address=145.79.227.6:4259 , piece_id=A73B6FI3W4257FJZ7PDYKCVOAK7FWXYOY5LAJO4MFF6WBBQEMHKQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:15.752349250 (578h55m15.623432285s) [noise=true , address=145.79.227.47:4266 , piece_id=QUMQLLHPAOBXGVT3EHGUO2PBHA3MVWSKQ6S3IISHUNECC7NU3OMA, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:15.756693101 (578h55m15.627776136s) [noise=true , address=145.79.227.11:4729 , piece_id=OJ2OYREMNIIWOOB63ZSSKUXI5L6GPJQBE7LFSGE766PFOYJQWVQA, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:15.759166110 (578h55m15.630249145s) [noise=true , address=145.79.227.36:4518 , piece_id=LJJIQM7QENYCHZXM7H2HE3TUNCXUWM453EZNH5HRMHXLGTDGQ4CA, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:15.765079070 (578h55m15.636162115s) [noise=true , address=145.79.227.50:4375 , piece_id=NA35IP4WGF5VPUPQFOKKPCK3BH5EKVEDP5WCHIHOC6OYISVNGAWQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:15.774799949 (578h55m15.645882984s) [noise=true , address=145.79.227.2:4113 , piece_id=KLRRZ6N6TIFNRMUO7QTAO6IEI7WOA2XWICPUQWXGV4QJKRHTR67Q, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:15.781963059 (578h55m15.653046094s) [noise=true , address=145.79.227.16:4909 , piece_id=XKGMGNTEIBSWHNHAUKAHR7D3DDOTO7QTBHE5KDHWH6UKNWIC6PBA, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:16.034054689 (578h55m15.905137734s) [noise=true , address=45.91.116.9:4264 , piece_id=QL2VPDIXIYTBKJ3LHTMXUQG47ANBBINESBZSUYJRCFS4N5HTBLHQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:16.088923118 (578h55m15.960006153s) [noise=true , address=45.91.116.8:4045 , piece_id=OMCXDTMQNC7CVXRXBHCS3NXHAEG5WHHDSKSF5FPJOGM2AWUZWHCA, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:16.222580781 (578h55m16.093663827s) [noise=true , address=45.91.116.14:4190 , piece_id=GWQK2PWNQQ3EOPGXNY3HITL526EPKZI2E6RKXV4F3OFQNT4VX6OQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:16.323827731 (578h55m16.194910766s) [noise=true , address=45.91.116.6:4974 , piece_id=F7GY2RNNW25WV7NK2M65VOJXUAGXFYZ62QB2OXZNEN3ANLKVY5MQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:16.323854903 (578h55m16.194937948s) [seg_pos={0 0}, upload=507499]: Segment reached optimal threshold of 13 pieces. Dec 29 18:24:16.324114357 (578h55m16.195197402s) [noise=true , address=45.91.116.7:4012 , piece_id=ZKE374IL4M5LW2KC7JNLZK7DFPBQPOYQMIPXALC2DJ3R7RUWBOFQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:50706->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:50706->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50706->45.91.116.7:4012: use of closed network connection Dec 29 18:24:16.324122292 (578h55m16.195205327s) [noise=true , address=45.91.116.7:4012 , piece_id=ZKE374IL4M5LW2KC7JNLZK7DFPBQPOYQMIPXALC2DJ3R7RUWBOFQ, seg_pos={0 0}, upload=507499]: Long tail cancellation due to optimal threshold Dec 29 18:24:16.324185493 (578h55m16.195268528s) [noise=true , address=77.90.31.11:4556 , piece_id=6VH5X3XLYBXCA4VPWYY33Z34XJ3WEDLHVLPX4B7XYFALFN6HPKOQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 29 18:24:16.324202294 (578h55m16.195285329s) [noise=true , address=77.90.31.11:4556 , piece_id=6VH5X3XLYBXCA4VPWYY33Z34XJ3WEDLHVLPX4B7XYFALFN6HPKOQ, seg_pos={0 0}, upload=507499]: Long tail cancellation due to optimal threshold Dec 29 18:24:16.324406734 (578h55m16.195489779s) [noise=true , address=77.90.31.14:4661 , piece_id=5HFUCZWSDRGXXKY2XF3J4TOK4P6VKPTZ56EIIR6CTADBGVUNDXEQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:59702->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:59702->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59702->77.90.31.14:4661: use of closed network connection Dec 29 18:24:16.324413406 (578h55m16.195496441s) [noise=true , address=77.90.31.14:4661 , piece_id=5HFUCZWSDRGXXKY2XF3J4TOK4P6VKPTZ56EIIR6CTADBGVUNDXEQ, seg_pos={0 0}, upload=507499]: Long tail cancellation due to optimal threshold Dec 29 18:24:16.324432272 (578h55m16.195515307s) [seg_pos={0 0}, upload=507499]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:24:16.324605092 (578h55m16.195688127s) [upload=507499]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:24:16.324610652 (578h55m16.195693697s) [upload=507499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:24:16.621668113 (578h55m16.492751148s) [upload=507499]: Done uploading segments... Dec 29 18:24:16.621691057 (578h55m16.492774092s) [upload=507499]: Upload finished. err: <nil> Duration: 30.003224206s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.420088409s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:24:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.420114379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 34.822215446s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:24:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 34.82224878s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.508063673s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:24:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.508161598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 27.61621003s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:23:37 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 27.616238933s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.541864925s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:23:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.541896274s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X......] duration: 6.359328271s trace: https://jaeger.dev.storj.io/trace/37deca3a5695bc8f FAIL (2025-12-29 18:23:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA) context deadline exceeded ======LOGS====== Dec 29 18:17:27.683505271 (578h48m27.554588307s) [upload=507405]: Starting upload Dec 29 18:17:27.683617325 (578h48m27.55470036s) [upload=507405]: Done starting upload Dec 29 18:17:27.683658934 (578h48m27.554741979s) [upload=507405]: Uploading segments... Dec 29 18:17:27.683903289 (578h48m27.554986334s) [upload=507405]: Got next segment. Inline: false Dec 29 18:17:27.683961059 (578h48m27.555044154s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:17:27.683990656 (578h48m27.555073721s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:17:27.928215687 (578h48m27.799298732s) [seg_pos={0 0}, upload=507405]: Begin upload segment... Dec 29 18:17:27.928360352 (578h48m27.799443397s) [seg_pos={0 0}, upload=507405]: Done begin upload segment. Dec 29 18:17:27.928406930 (578h48m27.799489965s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.928458439 (578h48m27.799541484s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.929193629 (578h48m27.800276674s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930052925 (578h48m27.80113597s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930621528 (578h48m27.801704583s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.932507930 (578h48m27.803590965s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.933327320 (578h48m27.804410365s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.935719154 (578h48m27.806802199s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.938347089 (578h48m27.809430124s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.940299085 (578h48m27.81138212s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.941852723 (578h48m27.812935768s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.943326871 (578h48m27.814409916s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.944782522 (578h48m27.815865567s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946438475 (578h48m27.817521521s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946869305 (578h48m27.817952341s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.949711929 (578h48m27.820794964s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:29.632497317 (578h48m29.503580352s) [upload=507405]: Next returned nil segment Dec 29 18:17:29.632515101 (578h48m29.503598176s) [upload=507405]: Waiting for error group managing segments... Dec 29 18:17:29.916807286 (578h48m29.787890331s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.926380194 (578h48m29.797463239s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.927162914 (578h48m29.798245959s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.932570482 (578h48m29.803653517s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.934594115 (578h48m29.80567715s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.939015764 (578h48m29.810098809s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.942541197 (578h48m29.813624232s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.944926970 (578h48m29.816010015s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.948098018 (578h48m29.819181053s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.949916971 (578h48m29.821000016s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.194442785 (578h48m30.06552583s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.289892726 (578h48m30.160975781s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528875687 (578h48m30.399958722s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528894383 (578h48m30.399977418s) [seg_pos={0 0}, upload=507405]: Segment reached optimal threshold of 13 pieces. Dec 29 18:17:30.529105565 (578h48m30.4001886s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529111987 (578h48m30.400195032s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529327357 (578h48m30.400410392s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529345903 (578h48m30.400428948s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529360069 (578h48m30.400443104s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529460661 (578h48m30.400543726s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529874990 (578h48m30.400958025s) [seg_pos={0 0}, upload=507405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:17:30.529955914 (578h48m30.401038949s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:17:30.529961635 (578h48m30.40104468s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:17:30.822776598 (578h48m30.693859643s) [upload=507405]: Done uploading segments... Dec 29 18:17:30.822800073 (578h48m30.693883118s) [upload=507405]: Upload finished. err: <nil> Duration: 6.359347457s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 36.676825899s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:22:31 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 36.676840997s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.587067693s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:22:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.587109974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 37.107171863s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:21:54 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 37.107188535s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X..XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X.....] duration: 5.727489623s trace: https://jaeger.dev.storj.io/trace/37deca3a5695bc8f FAIL (2025-12-29 18:21:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA) context deadline exceeded ======LOGS====== Dec 29 18:17:27.683505271 (578h48m27.554588307s) [upload=507405]: Starting upload Dec 29 18:17:27.683617325 (578h48m27.55470036s) [upload=507405]: Done starting upload Dec 29 18:17:27.683658934 (578h48m27.554741979s) [upload=507405]: Uploading segments... Dec 29 18:17:27.683903289 (578h48m27.554986334s) [upload=507405]: Got next segment. Inline: false Dec 29 18:17:27.683961059 (578h48m27.555044154s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:17:27.683990656 (578h48m27.555073721s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:17:27.928215687 (578h48m27.799298732s) [seg_pos={0 0}, upload=507405]: Begin upload segment... Dec 29 18:17:27.928360352 (578h48m27.799443397s) [seg_pos={0 0}, upload=507405]: Done begin upload segment. Dec 29 18:17:27.928406930 (578h48m27.799489965s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.928458439 (578h48m27.799541484s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.929193629 (578h48m27.800276674s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930052925 (578h48m27.80113597s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930621528 (578h48m27.801704583s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.932507930 (578h48m27.803590965s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.933327320 (578h48m27.804410365s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.935719154 (578h48m27.806802199s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.938347089 (578h48m27.809430124s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.940299085 (578h48m27.81138212s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.941852723 (578h48m27.812935768s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.943326871 (578h48m27.814409916s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.944782522 (578h48m27.815865567s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946438475 (578h48m27.817521521s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946869305 (578h48m27.817952341s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.949711929 (578h48m27.820794964s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:29.632497317 (578h48m29.503580352s) [upload=507405]: Next returned nil segment Dec 29 18:17:29.632515101 (578h48m29.503598176s) [upload=507405]: Waiting for error group managing segments... Dec 29 18:17:29.916807286 (578h48m29.787890331s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.926380194 (578h48m29.797463239s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.927162914 (578h48m29.798245959s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.932570482 (578h48m29.803653517s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.934594115 (578h48m29.80567715s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.939015764 (578h48m29.810098809s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.942541197 (578h48m29.813624232s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.944926970 (578h48m29.816010015s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.948098018 (578h48m29.819181053s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.949916971 (578h48m29.821000016s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.194442785 (578h48m30.06552583s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.289892726 (578h48m30.160975781s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528875687 (578h48m30.399958722s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528894383 (578h48m30.399977418s) [seg_pos={0 0}, upload=507405]: Segment reached optimal threshold of 13 pieces. Dec 29 18:17:30.529105565 (578h48m30.4001886s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529111987 (578h48m30.400195032s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529327357 (578h48m30.400410392s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529345903 (578h48m30.400428948s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529360069 (578h48m30.400443104s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529460661 (578h48m30.400543726s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529874990 (578h48m30.400958025s) [seg_pos={0 0}, upload=507405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:17:30.529955914 (578h48m30.401038949s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:17:30.529961635 (578h48m30.40104468s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:17:30.822776598 (578h48m30.693859643s) [upload=507405]: Done uploading segments... Dec 29 18:17:30.822800073 (578h48m30.693883118s) [upload=507405]: Upload finished. err: <nil> Duration: 5.727524489s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.456239945s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:21:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.456280611s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X..XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X....] duration: 8.741857335s trace: https://jaeger.dev.storj.io/trace/37deca3a5695bc8f FAIL (2025-12-29 18:21:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA) context deadline exceeded ======LOGS====== Dec 29 18:17:27.683505271 (578h48m27.554588307s) [upload=507405]: Starting upload Dec 29 18:17:27.683617325 (578h48m27.55470036s) [upload=507405]: Done starting upload Dec 29 18:17:27.683658934 (578h48m27.554741979s) [upload=507405]: Uploading segments... Dec 29 18:17:27.683903289 (578h48m27.554986334s) [upload=507405]: Got next segment. Inline: false Dec 29 18:17:27.683961059 (578h48m27.555044154s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:17:27.683990656 (578h48m27.555073721s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:17:27.928215687 (578h48m27.799298732s) [seg_pos={0 0}, upload=507405]: Begin upload segment... Dec 29 18:17:27.928360352 (578h48m27.799443397s) [seg_pos={0 0}, upload=507405]: Done begin upload segment. Dec 29 18:17:27.928406930 (578h48m27.799489965s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.928458439 (578h48m27.799541484s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.929193629 (578h48m27.800276674s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930052925 (578h48m27.80113597s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930621528 (578h48m27.801704583s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.932507930 (578h48m27.803590965s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.933327320 (578h48m27.804410365s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.935719154 (578h48m27.806802199s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.938347089 (578h48m27.809430124s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.940299085 (578h48m27.81138212s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.941852723 (578h48m27.812935768s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.943326871 (578h48m27.814409916s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.944782522 (578h48m27.815865567s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946438475 (578h48m27.817521521s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946869305 (578h48m27.817952341s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.949711929 (578h48m27.820794964s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:29.632497317 (578h48m29.503580352s) [upload=507405]: Next returned nil segment Dec 29 18:17:29.632515101 (578h48m29.503598176s) [upload=507405]: Waiting for error group managing segments... Dec 29 18:17:29.916807286 (578h48m29.787890331s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.926380194 (578h48m29.797463239s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.927162914 (578h48m29.798245959s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.932570482 (578h48m29.803653517s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.934594115 (578h48m29.80567715s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.939015764 (578h48m29.810098809s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.942541197 (578h48m29.813624232s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.944926970 (578h48m29.816010015s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.948098018 (578h48m29.819181053s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.949916971 (578h48m29.821000016s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.194442785 (578h48m30.06552583s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.289892726 (578h48m30.160975781s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528875687 (578h48m30.399958722s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528894383 (578h48m30.399977418s) [seg_pos={0 0}, upload=507405]: Segment reached optimal threshold of 13 pieces. Dec 29 18:17:30.529105565 (578h48m30.4001886s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529111987 (578h48m30.400195032s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529327357 (578h48m30.400410392s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529345903 (578h48m30.400428948s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529360069 (578h48m30.400443104s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529460661 (578h48m30.400543726s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529874990 (578h48m30.400958025s) [seg_pos={0 0}, upload=507405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:17:30.529955914 (578h48m30.401038949s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:17:30.529961635 (578h48m30.40104468s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:17:30.822776598 (578h48m30.693859643s) [upload=507405]: Done uploading segments... Dec 29 18:17:30.822800073 (578h48m30.693883118s) [upload=507405]: Upload finished. err: <nil> Duration: 8.741884737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 43.431571089s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:21:17 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 43.431603311s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.492615677s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:20:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.492711418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..X..XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X...] duration: 16.365733401s trace: https://jaeger.dev.storj.io/trace/37deca3a5695bc8f FAIL (2025-12-29 18:20:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA) context deadline exceeded ======LOGS====== Dec 29 18:17:27.683505271 (578h48m27.554588307s) [upload=507405]: Starting upload Dec 29 18:17:27.683617325 (578h48m27.55470036s) [upload=507405]: Done starting upload Dec 29 18:17:27.683658934 (578h48m27.554741979s) [upload=507405]: Uploading segments... Dec 29 18:17:27.683903289 (578h48m27.554986334s) [upload=507405]: Got next segment. Inline: false Dec 29 18:17:27.683961059 (578h48m27.555044154s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:17:27.683990656 (578h48m27.555073721s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:17:27.928215687 (578h48m27.799298732s) [seg_pos={0 0}, upload=507405]: Begin upload segment... Dec 29 18:17:27.928360352 (578h48m27.799443397s) [seg_pos={0 0}, upload=507405]: Done begin upload segment. Dec 29 18:17:27.928406930 (578h48m27.799489965s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.928458439 (578h48m27.799541484s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.929193629 (578h48m27.800276674s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930052925 (578h48m27.80113597s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930621528 (578h48m27.801704583s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.932507930 (578h48m27.803590965s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.933327320 (578h48m27.804410365s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.935719154 (578h48m27.806802199s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.938347089 (578h48m27.809430124s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.940299085 (578h48m27.81138212s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.941852723 (578h48m27.812935768s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.943326871 (578h48m27.814409916s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.944782522 (578h48m27.815865567s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946438475 (578h48m27.817521521s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946869305 (578h48m27.817952341s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.949711929 (578h48m27.820794964s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:29.632497317 (578h48m29.503580352s) [upload=507405]: Next returned nil segment Dec 29 18:17:29.632515101 (578h48m29.503598176s) [upload=507405]: Waiting for error group managing segments... Dec 29 18:17:29.916807286 (578h48m29.787890331s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.926380194 (578h48m29.797463239s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.927162914 (578h48m29.798245959s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.932570482 (578h48m29.803653517s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.934594115 (578h48m29.80567715s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.939015764 (578h48m29.810098809s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.942541197 (578h48m29.813624232s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.944926970 (578h48m29.816010015s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.948098018 (578h48m29.819181053s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.949916971 (578h48m29.821000016s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.194442785 (578h48m30.06552583s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.289892726 (578h48m30.160975781s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528875687 (578h48m30.399958722s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528894383 (578h48m30.399977418s) [seg_pos={0 0}, upload=507405]: Segment reached optimal threshold of 13 pieces. Dec 29 18:17:30.529105565 (578h48m30.4001886s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529111987 (578h48m30.400195032s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529327357 (578h48m30.400410392s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529345903 (578h48m30.400428948s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529360069 (578h48m30.400443104s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529460661 (578h48m30.400543726s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529874990 (578h48m30.400958025s) [seg_pos={0 0}, upload=507405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:17:30.529955914 (578h48m30.401038949s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:17:30.529961635 (578h48m30.40104468s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:17:30.822776598 (578h48m30.693859643s) [upload=507405]: Done uploading segments... Dec 29 18:17:30.822800073 (578h48m30.693883118s) [upload=507405]: Upload finished. err: <nil> Duration: 16.365763287s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 34.687822736s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:19:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 34.687840189s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.331934144s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:19:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.331960813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 35.991350282s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:19:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 35.991383404s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..X..XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X..] duration: 6.036358077s trace: https://jaeger.dev.storj.io/trace/37deca3a5695bc8f FAIL (2025-12-29 18:19:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA) context deadline exceeded ======LOGS====== Dec 29 18:17:27.683505271 (578h48m27.554588307s) [upload=507405]: Starting upload Dec 29 18:17:27.683617325 (578h48m27.55470036s) [upload=507405]: Done starting upload Dec 29 18:17:27.683658934 (578h48m27.554741979s) [upload=507405]: Uploading segments... Dec 29 18:17:27.683903289 (578h48m27.554986334s) [upload=507405]: Got next segment. Inline: false Dec 29 18:17:27.683961059 (578h48m27.555044154s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:17:27.683990656 (578h48m27.555073721s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:17:27.928215687 (578h48m27.799298732s) [seg_pos={0 0}, upload=507405]: Begin upload segment... Dec 29 18:17:27.928360352 (578h48m27.799443397s) [seg_pos={0 0}, upload=507405]: Done begin upload segment. Dec 29 18:17:27.928406930 (578h48m27.799489965s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.928458439 (578h48m27.799541484s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.929193629 (578h48m27.800276674s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930052925 (578h48m27.80113597s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930621528 (578h48m27.801704583s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.932507930 (578h48m27.803590965s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.933327320 (578h48m27.804410365s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.935719154 (578h48m27.806802199s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.938347089 (578h48m27.809430124s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.940299085 (578h48m27.81138212s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.941852723 (578h48m27.812935768s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.943326871 (578h48m27.814409916s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.944782522 (578h48m27.815865567s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946438475 (578h48m27.817521521s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946869305 (578h48m27.817952341s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.949711929 (578h48m27.820794964s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:29.632497317 (578h48m29.503580352s) [upload=507405]: Next returned nil segment Dec 29 18:17:29.632515101 (578h48m29.503598176s) [upload=507405]: Waiting for error group managing segments... Dec 29 18:17:29.916807286 (578h48m29.787890331s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.926380194 (578h48m29.797463239s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.927162914 (578h48m29.798245959s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.932570482 (578h48m29.803653517s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.934594115 (578h48m29.80567715s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.939015764 (578h48m29.810098809s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.942541197 (578h48m29.813624232s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.944926970 (578h48m29.816010015s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.948098018 (578h48m29.819181053s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.949916971 (578h48m29.821000016s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.194442785 (578h48m30.06552583s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.289892726 (578h48m30.160975781s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528875687 (578h48m30.399958722s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528894383 (578h48m30.399977418s) [seg_pos={0 0}, upload=507405]: Segment reached optimal threshold of 13 pieces. Dec 29 18:17:30.529105565 (578h48m30.4001886s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529111987 (578h48m30.400195032s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529327357 (578h48m30.400410392s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529345903 (578h48m30.400428948s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529360069 (578h48m30.400443104s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529460661 (578h48m30.400543726s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529874990 (578h48m30.400958025s) [seg_pos={0 0}, upload=507405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:17:30.529955914 (578h48m30.401038949s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:17:30.529961635 (578h48m30.40104468s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:17:30.822776598 (578h48m30.693859643s) [upload=507405]: Done uploading segments... Dec 29 18:17:30.822800073 (578h48m30.693883118s) [upload=507405]: Upload finished. err: <nil> Duration: 6.036377814s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)