Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [X.X...X........X.....X....XXX..X.X.X.......XX....X..X.XX....] duration: 3.597483253s trace: https://jaeger.dev.storj.io/trace/2a9d13babc34959 FAIL (2025-12-30 09:47:38 UTC) orbiter warning: context deadline exceeded Duration: 3.597515874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 47.918093299s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:47:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 47.918119008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X..XX..XX......X...X...X...X...X..............X....X.X..X.X.] duration: 5.976767657s trace: https://jaeger.dev.storj.io/trace/6f14fe4de74eb513 FAIL (2025-12-30 09:47:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: AAOFIGDMNL5REJVCEWUUHGWFB3IQGGD2GX5TZXAYZIS2BALRJISA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: P6YSNR5NSBDXSCHBOIWDQYB3OEMXXJY5ASPKLHDD3OIYMFC3YMEQ) context deadline exceeded; piecestore: (Node ID: 1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, Piece ID: FUQFOTOIAVOPINQIY3W4CPY72M4QV4ZCX265V5E74ZRWKR7RSHEA) context deadline exceeded ======LOGS====== Dec 30 09:45:54.941531524 (594h16m54.812614559s) [upload=520962]: Starting upload Dec 30 09:45:54.941589874 (594h16m54.812672919s) [upload=520962]: Done starting upload Dec 30 09:45:54.941696047 (594h16m54.812779082s) [upload=520962]: Uploading segments... Dec 30 09:45:54.941830342 (594h16m54.812913377s) [upload=520962]: Got next segment. Inline: false Dec 30 09:45:54.941864998 (594h16m54.812948033s) [upload=520962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:45:54.941870439 (594h16m54.812953484s) [upload=520962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:45:55.188150120 (594h16m55.059233155s) [seg_pos={0 0}, upload=520962]: Begin upload segment... Dec 30 09:45:55.188208361 (594h16m55.059291396s) [seg_pos={0 0}, upload=520962]: Done begin upload segment. Dec 30 09:45:55.188236093 (594h16m55.059319128s) [noise=true , address=145.79.227.31:4799 , piece_id=4YDYJQUBL6H45E2YE5BUXY433N5ZRZ4AKYSP5UPIC6XZB2C6UF2Q, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.190766401 (594h16m55.061849437s) [noise=true , address=145.79.227.23:4545 , piece_id=LWN2PS56VYYJA5XWNFJNJLJYNVX5ANZODKXO4DQOSXFLCM47NEKA, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.191205067 (594h16m55.062288102s) [noise=true , address=45.91.116.5:4297 , piece_id=AAOFIGDMNL5REJVCEWUUHGWFB3IQGGD2GX5TZXAYZIS2BALRJISA, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.191559843 (594h16m55.062642878s) [noise=true , address=45.91.116.7:4009 , piece_id=4JYAJMPWOKFLVBPTJ64UVGMK7AO77HKKSQB4TEPNPVOECLEAPFRQ, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.191886033 (594h16m55.062969068s) [noise=true , address=45.91.116.8:4045 , piece_id=G5UM3TZBIAVSA57TSVYQKZJCOPGVFAIDJQ3TRJYWD2JM6OVEN5CA, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.192177659 (594h16m55.063260694s) [noise=true , address=45.91.116.12:4227 , piece_id=P6YSNR5NSBDXSCHBOIWDQYB3OEMXXJY5ASPKLHDD3OIYMFC3YMEQ, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.192531202 (594h16m55.063614237s) [noise=true , address=145.79.227.26:4654 , piece_id=FUQFOTOIAVOPINQIY3W4CPY72M4QV4ZCX265V5E74ZRWKR7RSHEA, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.192885697 (594h16m55.063968732s) [noise=true , address=145.79.227.8:4331 , piece_id=NPJZ2PDKQ75IJSLEEVLCXXBZT4YHJC5HWUQ4WHGB3PYLQ6Y6KYFA, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.193273525 (594h16m55.064356571s) [noise=true , address=145.79.227.35:4481 , piece_id=OBFCWZ6LRTPORWUFT6LL4XDSNF4MOJYVWL4B43BNAO5VYOY7CSKQ, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.193648560 (594h16m55.064731595s) [noise=true , address=145.79.227.4:4185 , piece_id=TSGAHRL2O6MWKQ43HEODKEZ754U7EZ4Y4QAEMT3FSYF74NDAMV4Q, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.195007888 (594h16m55.066090923s) [noise=true , address=145.79.227.42:4122 , piece_id=BBNLWPEKYIZ7IYB5DS7J3HU2VLZTFQGBV5NK7EBKQPIS4WWV4A5Q, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.195769709 (594h16m55.066852754s) [noise=true , address=145.79.227.38:4977 , piece_id=5MHEJZ2XADRIZUWA34FCQEPDZGYSO6PIAAQLYC4IQ5YUNK7OPMOA, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.196247128 (594h16m55.067330163s) [noise=true , address=145.79.227.7:4293 , piece_id=OIEJPGKYT4VA24L3HF4U3NZBCZHGOGDHV2UORLBJSMLUKYKCN3CQ, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.196701584 (594h16m55.067784619s) [noise=true , address=145.79.227.13:4804 , piece_id=AE3J4JDF63QOTC7PBIBC3PH2XUGQSQJFOPBI2QNVU7TBRUY2EDXQ, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.197200343 (594h16m55.068283378s) [noise=true , address=145.79.227.11:4732 , piece_id=M2TNPOXLE4T25TE3OM3RYBWU4DBUIZGSGJUUNW42BVKXLKZBNTKQ, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.197560909 (594h16m55.068643944s) [noise=true , address=45.91.116.6:4975 , piece_id=F4LKZ5AACJHXBHNQUX2MD3B64DEE4MMZDN44RXAHUJESFVQXRFNA, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:56.800840511 (594h16m56.671923556s) [upload=520962]: Next returned nil segment Dec 30 09:45:56.800859266 (594h16m56.671942312s) [upload=520962]: Waiting for error group managing segments... Dec 30 09:45:57.180793620 (594h16m57.051876655s) [noise=true , address=145.79.227.8:4331 , piece_id=NPJZ2PDKQ75IJSLEEVLCXXBZT4YHJC5HWUQ4WHGB3PYLQ6Y6KYFA, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.180977029 (594h16m57.052060064s) [noise=true , address=145.79.227.13:4804 , piece_id=AE3J4JDF63QOTC7PBIBC3PH2XUGQSQJFOPBI2QNVU7TBRUY2EDXQ, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.182409035 (594h16m57.05349208s) [noise=true , address=145.79.227.23:4545 , piece_id=LWN2PS56VYYJA5XWNFJNJLJYNVX5ANZODKXO4DQOSXFLCM47NEKA, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.184429493 (594h16m57.055512528s) [noise=true , address=145.79.227.38:4977 , piece_id=5MHEJZ2XADRIZUWA34FCQEPDZGYSO6PIAAQLYC4IQ5YUNK7OPMOA, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.184776303 (594h16m57.055859348s) [noise=true , address=145.79.227.26:4654 , piece_id=FUQFOTOIAVOPINQIY3W4CPY72M4QV4ZCX265V5E74ZRWKR7RSHEA, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.189334102 (594h16m57.060417137s) [noise=true , address=145.79.227.42:4122 , piece_id=BBNLWPEKYIZ7IYB5DS7J3HU2VLZTFQGBV5NK7EBKQPIS4WWV4A5Q, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.205424692 (594h16m57.076507727s) [noise=true , address=145.79.227.31:4799 , piece_id=4YDYJQUBL6H45E2YE5BUXY433N5ZRZ4AKYSP5UPIC6XZB2C6UF2Q, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.211881306 (594h16m57.082964341s) [noise=true , address=145.79.227.4:4185 , piece_id=TSGAHRL2O6MWKQ43HEODKEZ754U7EZ4Y4QAEMT3FSYF74NDAMV4Q, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.216035707 (594h16m57.087118742s) [noise=true , address=145.79.227.11:4732 , piece_id=M2TNPOXLE4T25TE3OM3RYBWU4DBUIZGSGJUUNW42BVKXLKZBNTKQ, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.216935862 (594h16m57.088018907s) [noise=true , address=145.79.227.7:4293 , piece_id=OIEJPGKYT4VA24L3HF4U3NZBCZHGOGDHV2UORLBJSMLUKYKCN3CQ, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.489351249 (594h16m57.360434284s) [noise=true , address=45.91.116.7:4009 , piece_id=4JYAJMPWOKFLVBPTJ64UVGMK7AO77HKKSQB4TEPNPVOECLEAPFRQ, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.620587466 (594h16m57.491670511s) [noise=true , address=45.91.116.12:4227 , piece_id=P6YSNR5NSBDXSCHBOIWDQYB3OEMXXJY5ASPKLHDD3OIYMFC3YMEQ, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.777689968 (594h16m57.648773013s) [noise=true , address=45.91.116.5:4297 , piece_id=AAOFIGDMNL5REJVCEWUUHGWFB3IQGGD2GX5TZXAYZIS2BALRJISA, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.777709063 (594h16m57.648792098s) [seg_pos={0 0}, upload=520962]: Segment reached optimal threshold of 13 pieces. Dec 30 09:45:57.777894647 (594h16m57.648977682s) [noise=true , address=145.79.227.35:4481 , piece_id=OBFCWZ6LRTPORWUFT6LL4XDSNF4MOJYVWL4B43BNAO5VYOY7CSKQ, seg_pos={0 0}, upload=520962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 09:45:57.777901220 (594h16m57.648984255s) [noise=true , address=145.79.227.35:4481 , piece_id=OBFCWZ6LRTPORWUFT6LL4XDSNF4MOJYVWL4B43BNAO5VYOY7CSKQ, seg_pos={0 0}, upload=520962]: Long tail cancellation due to optimal threshold Dec 30 09:45:57.778007763 (594h16m57.649090798s) [noise=true , address=45.91.116.8:4045 , piece_id=G5UM3TZBIAVSA57TSVYQKZJCOPGVFAIDJQ3TRJYWD2JM6OVEN5CA, seg_pos={0 0}, upload=520962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 30 09:45:57.778012682 (594h16m57.649095727s) [noise=true , address=45.91.116.8:4045 , piece_id=G5UM3TZBIAVSA57TSVYQKZJCOPGVFAIDJQ3TRJYWD2JM6OVEN5CA, seg_pos={0 0}, upload=520962]: Long tail cancellation due to optimal threshold Dec 30 09:45:57.778109687 (594h16m57.649192732s) [noise=true , address=45.91.116.6:4975 , piece_id=F4LKZ5AACJHXBHNQUX2MD3B64DEE4MMZDN44RXAHUJESFVQXRFNA, seg_pos={0 0}, upload=520962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 09:45:57.778115468 (594h16m57.649198503s) [noise=true , address=45.91.116.6:4975 , piece_id=F4LKZ5AACJHXBHNQUX2MD3B64DEE4MMZDN44RXAHUJESFVQXRFNA, seg_pos={0 0}, upload=520962]: Long tail cancellation due to optimal threshold Dec 30 09:45:57.778129955 (594h16m57.64921299s) [seg_pos={0 0}, upload=520962]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:45:57.778199066 (594h16m57.649282101s) [upload=520962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:45:57.778203846 (594h16m57.649286881s) [upload=520962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:45:58.065674462 (594h16m57.936757517s) [upload=520962]: Done uploading segments... Dec 30 09:45:58.065696523 (594h16m57.936779568s) [upload=520962]: Upload finished. err: <nil> Duration: 5.976796902s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.X.X...X........X.....X....XXX..X.X.X.......XX....X..X.XX...] duration: 3.558569724s trace: https://jaeger.dev.storj.io/trace/2a9d13babc34959 FAIL (2025-12-30 09:46:48 UTC) orbiter warning: context deadline exceeded Duration: 3.558597137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 40.617983859s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:46:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 40.618001462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X..XX..XX......X...X...X...X...X..............X....X.X..X.X] duration: 30.00377837s trace: https://jaeger.dev.storj.io/trace/6f14fe4de74eb513 FAIL (2025-12-30 09:46:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: AAOFIGDMNL5REJVCEWUUHGWFB3IQGGD2GX5TZXAYZIS2BALRJISA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: P6YSNR5NSBDXSCHBOIWDQYB3OEMXXJY5ASPKLHDD3OIYMFC3YMEQ) context deadline exceeded; piecestore: (Node ID: 1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, Piece ID: FUQFOTOIAVOPINQIY3W4CPY72M4QV4ZCX265V5E74ZRWKR7RSHEA) context deadline exceeded ======LOGS====== Dec 30 09:45:54.941531524 (594h16m54.812614559s) [upload=520962]: Starting upload Dec 30 09:45:54.941589874 (594h16m54.812672919s) [upload=520962]: Done starting upload Dec 30 09:45:54.941696047 (594h16m54.812779082s) [upload=520962]: Uploading segments... Dec 30 09:45:54.941830342 (594h16m54.812913377s) [upload=520962]: Got next segment. Inline: false Dec 30 09:45:54.941864998 (594h16m54.812948033s) [upload=520962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:45:54.941870439 (594h16m54.812953484s) [upload=520962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:45:55.188150120 (594h16m55.059233155s) [seg_pos={0 0}, upload=520962]: Begin upload segment... Dec 30 09:45:55.188208361 (594h16m55.059291396s) [seg_pos={0 0}, upload=520962]: Done begin upload segment. Dec 30 09:45:55.188236093 (594h16m55.059319128s) [noise=true , address=145.79.227.31:4799 , piece_id=4YDYJQUBL6H45E2YE5BUXY433N5ZRZ4AKYSP5UPIC6XZB2C6UF2Q, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.190766401 (594h16m55.061849437s) [noise=true , address=145.79.227.23:4545 , piece_id=LWN2PS56VYYJA5XWNFJNJLJYNVX5ANZODKXO4DQOSXFLCM47NEKA, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.191205067 (594h16m55.062288102s) [noise=true , address=45.91.116.5:4297 , piece_id=AAOFIGDMNL5REJVCEWUUHGWFB3IQGGD2GX5TZXAYZIS2BALRJISA, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.191559843 (594h16m55.062642878s) [noise=true , address=45.91.116.7:4009 , piece_id=4JYAJMPWOKFLVBPTJ64UVGMK7AO77HKKSQB4TEPNPVOECLEAPFRQ, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.191886033 (594h16m55.062969068s) [noise=true , address=45.91.116.8:4045 , piece_id=G5UM3TZBIAVSA57TSVYQKZJCOPGVFAIDJQ3TRJYWD2JM6OVEN5CA, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.192177659 (594h16m55.063260694s) [noise=true , address=45.91.116.12:4227 , piece_id=P6YSNR5NSBDXSCHBOIWDQYB3OEMXXJY5ASPKLHDD3OIYMFC3YMEQ, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.192531202 (594h16m55.063614237s) [noise=true , address=145.79.227.26:4654 , piece_id=FUQFOTOIAVOPINQIY3W4CPY72M4QV4ZCX265V5E74ZRWKR7RSHEA, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.192885697 (594h16m55.063968732s) [noise=true , address=145.79.227.8:4331 , piece_id=NPJZ2PDKQ75IJSLEEVLCXXBZT4YHJC5HWUQ4WHGB3PYLQ6Y6KYFA, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.193273525 (594h16m55.064356571s) [noise=true , address=145.79.227.35:4481 , piece_id=OBFCWZ6LRTPORWUFT6LL4XDSNF4MOJYVWL4B43BNAO5VYOY7CSKQ, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.193648560 (594h16m55.064731595s) [noise=true , address=145.79.227.4:4185 , piece_id=TSGAHRL2O6MWKQ43HEODKEZ754U7EZ4Y4QAEMT3FSYF74NDAMV4Q, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.195007888 (594h16m55.066090923s) [noise=true , address=145.79.227.42:4122 , piece_id=BBNLWPEKYIZ7IYB5DS7J3HU2VLZTFQGBV5NK7EBKQPIS4WWV4A5Q, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.195769709 (594h16m55.066852754s) [noise=true , address=145.79.227.38:4977 , piece_id=5MHEJZ2XADRIZUWA34FCQEPDZGYSO6PIAAQLYC4IQ5YUNK7OPMOA, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.196247128 (594h16m55.067330163s) [noise=true , address=145.79.227.7:4293 , piece_id=OIEJPGKYT4VA24L3HF4U3NZBCZHGOGDHV2UORLBJSMLUKYKCN3CQ, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.196701584 (594h16m55.067784619s) [noise=true , address=145.79.227.13:4804 , piece_id=AE3J4JDF63QOTC7PBIBC3PH2XUGQSQJFOPBI2QNVU7TBRUY2EDXQ, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.197200343 (594h16m55.068283378s) [noise=true , address=145.79.227.11:4732 , piece_id=M2TNPOXLE4T25TE3OM3RYBWU4DBUIZGSGJUUNW42BVKXLKZBNTKQ, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:55.197560909 (594h16m55.068643944s) [noise=true , address=45.91.116.6:4975 , piece_id=F4LKZ5AACJHXBHNQUX2MD3B64DEE4MMZDN44RXAHUJESFVQXRFNA, seg_pos={0 0}, upload=520962]: Uploading piece... Dec 30 09:45:56.800840511 (594h16m56.671923556s) [upload=520962]: Next returned nil segment Dec 30 09:45:56.800859266 (594h16m56.671942312s) [upload=520962]: Waiting for error group managing segments... Dec 30 09:45:57.180793620 (594h16m57.051876655s) [noise=true , address=145.79.227.8:4331 , piece_id=NPJZ2PDKQ75IJSLEEVLCXXBZT4YHJC5HWUQ4WHGB3PYLQ6Y6KYFA, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.180977029 (594h16m57.052060064s) [noise=true , address=145.79.227.13:4804 , piece_id=AE3J4JDF63QOTC7PBIBC3PH2XUGQSQJFOPBI2QNVU7TBRUY2EDXQ, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.182409035 (594h16m57.05349208s) [noise=true , address=145.79.227.23:4545 , piece_id=LWN2PS56VYYJA5XWNFJNJLJYNVX5ANZODKXO4DQOSXFLCM47NEKA, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.184429493 (594h16m57.055512528s) [noise=true , address=145.79.227.38:4977 , piece_id=5MHEJZ2XADRIZUWA34FCQEPDZGYSO6PIAAQLYC4IQ5YUNK7OPMOA, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.184776303 (594h16m57.055859348s) [noise=true , address=145.79.227.26:4654 , piece_id=FUQFOTOIAVOPINQIY3W4CPY72M4QV4ZCX265V5E74ZRWKR7RSHEA, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.189334102 (594h16m57.060417137s) [noise=true , address=145.79.227.42:4122 , piece_id=BBNLWPEKYIZ7IYB5DS7J3HU2VLZTFQGBV5NK7EBKQPIS4WWV4A5Q, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.205424692 (594h16m57.076507727s) [noise=true , address=145.79.227.31:4799 , piece_id=4YDYJQUBL6H45E2YE5BUXY433N5ZRZ4AKYSP5UPIC6XZB2C6UF2Q, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.211881306 (594h16m57.082964341s) [noise=true , address=145.79.227.4:4185 , piece_id=TSGAHRL2O6MWKQ43HEODKEZ754U7EZ4Y4QAEMT3FSYF74NDAMV4Q, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.216035707 (594h16m57.087118742s) [noise=true , address=145.79.227.11:4732 , piece_id=M2TNPOXLE4T25TE3OM3RYBWU4DBUIZGSGJUUNW42BVKXLKZBNTKQ, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.216935862 (594h16m57.088018907s) [noise=true , address=145.79.227.7:4293 , piece_id=OIEJPGKYT4VA24L3HF4U3NZBCZHGOGDHV2UORLBJSMLUKYKCN3CQ, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.489351249 (594h16m57.360434284s) [noise=true , address=45.91.116.7:4009 , piece_id=4JYAJMPWOKFLVBPTJ64UVGMK7AO77HKKSQB4TEPNPVOECLEAPFRQ, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.620587466 (594h16m57.491670511s) [noise=true , address=45.91.116.12:4227 , piece_id=P6YSNR5NSBDXSCHBOIWDQYB3OEMXXJY5ASPKLHDD3OIYMFC3YMEQ, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.777689968 (594h16m57.648773013s) [noise=true , address=45.91.116.5:4297 , piece_id=AAOFIGDMNL5REJVCEWUUHGWFB3IQGGD2GX5TZXAYZIS2BALRJISA, seg_pos={0 0}, upload=520962]: Done uploading piece. err: <nil> Dec 30 09:45:57.777709063 (594h16m57.648792098s) [seg_pos={0 0}, upload=520962]: Segment reached optimal threshold of 13 pieces. Dec 30 09:45:57.777894647 (594h16m57.648977682s) [noise=true , address=145.79.227.35:4481 , piece_id=OBFCWZ6LRTPORWUFT6LL4XDSNF4MOJYVWL4B43BNAO5VYOY7CSKQ, seg_pos={0 0}, upload=520962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 09:45:57.777901220 (594h16m57.648984255s) [noise=true , address=145.79.227.35:4481 , piece_id=OBFCWZ6LRTPORWUFT6LL4XDSNF4MOJYVWL4B43BNAO5VYOY7CSKQ, seg_pos={0 0}, upload=520962]: Long tail cancellation due to optimal threshold Dec 30 09:45:57.778007763 (594h16m57.649090798s) [noise=true , address=45.91.116.8:4045 , piece_id=G5UM3TZBIAVSA57TSVYQKZJCOPGVFAIDJQ3TRJYWD2JM6OVEN5CA, seg_pos={0 0}, upload=520962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 30 09:45:57.778012682 (594h16m57.649095727s) [noise=true , address=45.91.116.8:4045 , piece_id=G5UM3TZBIAVSA57TSVYQKZJCOPGVFAIDJQ3TRJYWD2JM6OVEN5CA, seg_pos={0 0}, upload=520962]: Long tail cancellation due to optimal threshold Dec 30 09:45:57.778109687 (594h16m57.649192732s) [noise=true , address=45.91.116.6:4975 , piece_id=F4LKZ5AACJHXBHNQUX2MD3B64DEE4MMZDN44RXAHUJESFVQXRFNA, seg_pos={0 0}, upload=520962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 09:45:57.778115468 (594h16m57.649198503s) [noise=true , address=45.91.116.6:4975 , piece_id=F4LKZ5AACJHXBHNQUX2MD3B64DEE4MMZDN44RXAHUJESFVQXRFNA, seg_pos={0 0}, upload=520962]: Long tail cancellation due to optimal threshold Dec 30 09:45:57.778129955 (594h16m57.64921299s) [seg_pos={0 0}, upload=520962]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:45:57.778199066 (594h16m57.649282101s) [upload=520962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:45:57.778203846 (594h16m57.649286881s) [upload=520962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:45:58.065674462 (594h16m57.936757517s) [upload=520962]: Done uploading segments... Dec 30 09:45:58.065696523 (594h16m57.936779568s) [upload=520962]: Upload finished. err: <nil> Duration: 30.003798318s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [..X.X...X........X.....X....XXX..X.X.X.......XX....X..X.XX..] duration: 3.498664443s trace: https://jaeger.dev.storj.io/trace/2a9d13babc34959 FAIL (2025-12-30 09:45:52 UTC) orbiter warning: context deadline exceeded Duration: 3.498691784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 47.720827805s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:45:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 47.720855988s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X..XX..XX......X...X...X...X...X..............X....X.X..X.] duration: 5.761108953s trace: https://jaeger.dev.storj.io/trace/29d2306556baccd3 FAIL (2025-12-30 09:45:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 4YENLUEZB52PFVDBHPOX2EHLJ74IRI42KYEVZPHBQMODUYBUQEJA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 3ALOT7M65USCE2SPDMMYMUH7ZABAROLYQL2TXGSQNAIIVSVO266Q) context deadline exceeded ======LOGS====== Dec 30 09:44:57.622527623 (594h15m57.493610668s) [upload=520946]: Starting upload Dec 30 09:44:57.622624046 (594h15m57.493707112s) [upload=520946]: Done starting upload Dec 30 09:44:57.622764704 (594h15m57.493847749s) [upload=520946]: Uploading segments... Dec 30 09:44:57.623089512 (594h15m57.494172558s) [upload=520946]: Got next segment. Inline: false Dec 30 09:44:57.623147022 (594h15m57.494230067s) [upload=520946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:44:57.623173563 (594h15m57.494256628s) [upload=520946]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:44:57.871673551 (594h15m57.742756596s) [seg_pos={0 0}, upload=520946]: Begin upload segment... Dec 30 09:44:57.871800984 (594h15m57.742884019s) [noise=true , address=45.91.116.9:4261 , piece_id=RWRQJOBRBFVPSJJSO5GEE5HGAUC5YOMNGSN7NK4PYRRP2ID34ZDA, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.871939869 (594h15m57.743022914s) [seg_pos={0 0}, upload=520946]: Done begin upload segment. Dec 30 09:44:57.872155219 (594h15m57.743238264s) [noise=true , address=45.91.116.12:4225 , piece_id=3ALOT7M65USCE2SPDMMYMUH7ZABAROLYQL2TXGSQNAIIVSVO266Q, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.874133145 (594h15m57.74521619s) [noise=true , address=145.79.227.19:4079 , piece_id=EWSYXMKBNBLJDOKGT5AIK5B3PX6Q7DGDEUGKU6B3KOCACO5YQZWQ, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.874996529 (594h15m57.746079605s) [noise=true , address=145.79.227.3:4149 , piece_id=4OMMF2SOZRL2X3FQEPG36A4TDZ5Y6ML7E2VU3QZLGO6BIZTUX5JQ, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.877030292 (594h15m57.748113327s) [noise=true , address=145.79.227.51:4410 , piece_id=SUXJYQ6LTZXHUIWPICDSZW2DWHQLBPSPACTX2566MZZUF32NT2KQ, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.877499646 (594h15m57.748582681s) [noise=true , address=145.79.227.50:4374 , piece_id=NFJ7DGXIBSCWHYGWBA6V7IHAGUY3JLITESB3ROVDYGNXH2APH7LA, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.881845311 (594h15m57.752928356s) [noise=true , address=145.79.227.25:4617 , piece_id=HDSEJJ5PWH24AH7SVTU33SWPFH5ETXUFCQVMSMIZDSXOVHQJL53Q, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.884276821 (594h15m57.755359856s) [noise=true , address=145.79.227.33:4870 , piece_id=WO4MUHFRHDNZQFBN7NP7E77KGL3XQ222MVNXV2JZZUI2ZSYZEV3Q, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.884588313 (594h15m57.755671348s) [noise=true , address=145.79.227.15:4874 , piece_id=EPT5WTJK4UDSWXPAYTROUQTJUCOS4V2AUXRIOXE27E3Y3D2ZOLTA, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.885959605 (594h15m57.75704264s) [noise=true , address=145.79.227.41:4087 , piece_id=C5PQ7VHX3GGDGGVT4QLBZZ5AT2IW2FJ4PEEGAAYHSQKKHREW7KYQ, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.887080730 (594h15m57.758163775s) [noise=true , address=145.79.227.52:4447 , piece_id=O6QCURQEPHEHW45KK3PV7FT573BWDBZL6Y5WMOEZXKEXFMTIWMGQ, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.888486356 (594h15m57.759569401s) [noise=true , address=77.90.31.15:4700 , piece_id=KAO3KG4G6DTVOA5WUSZ5FDH4ZPF57UPPMBZRLMQX5J6IIMNXDR5A, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.889751465 (594h15m57.760834501s) [noise=true , address=145.79.227.47:4267 , piece_id=NNMMR7KSQTKFQSMY2A6XWZVNYV2WEVH7MBYEEZHP3YLEDEMIYUWQ, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.890821583 (594h15m57.761904618s) [noise=true , address=145.79.227.31:4799 , piece_id=HGFAUEFCHYUT652V3SVXIRK4HFDTZQ7PRLU3DTMLSWXW6DJ5KHCQ, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.892028371 (594h15m57.763111406s) [noise=true , address=77.90.31.14:4663 , piece_id=4YENLUEZB52PFVDBHPOX2EHLJ74IRI42KYEVZPHBQMODUYBUQEJA, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.894879750 (594h15m57.765962785s) [noise=true , address=145.79.227.2:4115 , piece_id=DRVLXSM4QVGYXU7GDRV3JFVWGJCSHIJ4DPOELGJOLMMCZ7FET33Q, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:59.588369427 (594h15m59.459452473s) [upload=520946]: Next returned nil segment Dec 30 09:44:59.588399084 (594h15m59.459482119s) [upload=520946]: Waiting for error group managing segments... Dec 30 09:44:59.869084860 (594h15m59.740167895s) [noise=true , address=145.79.227.50:4374 , piece_id=NFJ7DGXIBSCWHYGWBA6V7IHAGUY3JLITESB3ROVDYGNXH2APH7LA, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.877499113 (594h15m59.748582148s) [noise=true , address=145.79.227.3:4149 , piece_id=4OMMF2SOZRL2X3FQEPG36A4TDZ5Y6ML7E2VU3QZLGO6BIZTUX5JQ, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.878361345 (594h15m59.74944438s) [noise=true , address=145.79.227.25:4617 , piece_id=HDSEJJ5PWH24AH7SVTU33SWPFH5ETXUFCQVMSMIZDSXOVHQJL53Q, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.896653025 (594h15m59.76773606s) [noise=true , address=145.79.227.2:4115 , piece_id=DRVLXSM4QVGYXU7GDRV3JFVWGJCSHIJ4DPOELGJOLMMCZ7FET33Q, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.897270090 (594h15m59.768353165s) [noise=true , address=145.79.227.52:4447 , piece_id=O6QCURQEPHEHW45KK3PV7FT573BWDBZL6Y5WMOEZXKEXFMTIWMGQ, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.908624811 (594h15m59.779707856s) [noise=true , address=145.79.227.41:4087 , piece_id=C5PQ7VHX3GGDGGVT4QLBZZ5AT2IW2FJ4PEEGAAYHSQKKHREW7KYQ, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.920493381 (594h15m59.791576426s) [noise=true , address=145.79.227.31:4799 , piece_id=HGFAUEFCHYUT652V3SVXIRK4HFDTZQ7PRLU3DTMLSWXW6DJ5KHCQ, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.933222740 (594h15m59.804305775s) [noise=true , address=145.79.227.47:4267 , piece_id=NNMMR7KSQTKFQSMY2A6XWZVNYV2WEVH7MBYEEZHP3YLEDEMIYUWQ, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.938872648 (594h15m59.809955683s) [noise=true , address=145.79.227.19:4079 , piece_id=EWSYXMKBNBLJDOKGT5AIK5B3PX6Q7DGDEUGKU6B3KOCACO5YQZWQ, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.961010433 (594h15m59.832093478s) [noise=true , address=145.79.227.51:4410 , piece_id=SUXJYQ6LTZXHUIWPICDSZW2DWHQLBPSPACTX2566MZZUF32NT2KQ, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.980899836 (594h15m59.851982871s) [noise=true , address=145.79.227.15:4874 , piece_id=EPT5WTJK4UDSWXPAYTROUQTJUCOS4V2AUXRIOXE27E3Y3D2ZOLTA, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:45:00.183953407 (594h16m0.055036442s) [noise=true , address=77.90.31.14:4663 , piece_id=4YENLUEZB52PFVDBHPOX2EHLJ74IRI42KYEVZPHBQMODUYBUQEJA, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:45:00.309021879 (594h16m0.180104914s) [noise=true , address=45.91.116.12:4225 , piece_id=3ALOT7M65USCE2SPDMMYMUH7ZABAROLYQL2TXGSQNAIIVSVO266Q, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:45:00.309041327 (594h16m0.180124362s) [seg_pos={0 0}, upload=520946]: Segment reached optimal threshold of 13 pieces. Dec 30 09:45:00.309267126 (594h16m0.180350161s) [noise=true , address=45.91.116.9:4261 , piece_id=RWRQJOBRBFVPSJJSO5GEE5HGAUC5YOMNGSN7NK4PYRRP2ID34ZDA, seg_pos={0 0}, upload=520946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 30 09:45:00.309274149 (594h16m0.180357184s) [noise=true , address=45.91.116.9:4261 , piece_id=RWRQJOBRBFVPSJJSO5GEE5HGAUC5YOMNGSN7NK4PYRRP2ID34ZDA, seg_pos={0 0}, upload=520946]: Long tail cancellation due to optimal threshold Dec 30 09:45:00.309448141 (594h16m0.180531176s) [noise=true , address=77.90.31.15:4700 , piece_id=KAO3KG4G6DTVOA5WUSZ5FDH4ZPF57UPPMBZRLMQX5J6IIMNXDR5A, seg_pos={0 0}, upload=520946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 30 09:45:00.309469021 (594h16m0.180552056s) [noise=true , address=77.90.31.15:4700 , piece_id=KAO3KG4G6DTVOA5WUSZ5FDH4ZPF57UPPMBZRLMQX5J6IIMNXDR5A, seg_pos={0 0}, upload=520946]: Long tail cancellation due to optimal threshold Dec 30 09:45:00.309503036 (594h16m0.180586071s) [noise=true , address=145.79.227.33:4870 , piece_id=WO4MUHFRHDNZQFBN7NP7E77KGL3XQ222MVNXV2JZZUI2ZSYZEV3Q, seg_pos={0 0}, upload=520946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 09:45:00.309509938 (594h16m0.180592984s) [noise=true , address=145.79.227.33:4870 , piece_id=WO4MUHFRHDNZQFBN7NP7E77KGL3XQ222MVNXV2JZZUI2ZSYZEV3Q, seg_pos={0 0}, upload=520946]: Long tail cancellation due to optimal threshold Dec 30 09:45:00.309553351 (594h16m0.180636396s) [seg_pos={0 0}, upload=520946]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:45:00.309627412 (594h16m0.180710447s) [upload=520946]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:45:00.309633715 (594h16m0.18071676s) [upload=520946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:45:00.607168544 (594h16m0.478251589s) [upload=520946]: Done uploading segments... Dec 30 09:45:00.607191698 (594h16m0.478274743s) [upload=520946]: Upload finished. err: <nil> Duration: 5.761124133s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X..XX..XX......X...X...X...X...X..............X....X.X..X] duration: 30.002202309s trace: https://jaeger.dev.storj.io/trace/29d2306556baccd3 FAIL (2025-12-30 09:45:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 4YENLUEZB52PFVDBHPOX2EHLJ74IRI42KYEVZPHBQMODUYBUQEJA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 3ALOT7M65USCE2SPDMMYMUH7ZABAROLYQL2TXGSQNAIIVSVO266Q) context deadline exceeded ======LOGS====== Dec 30 09:44:57.622527623 (594h15m57.493610668s) [upload=520946]: Starting upload Dec 30 09:44:57.622624046 (594h15m57.493707112s) [upload=520946]: Done starting upload Dec 30 09:44:57.622764704 (594h15m57.493847749s) [upload=520946]: Uploading segments... Dec 30 09:44:57.623089512 (594h15m57.494172558s) [upload=520946]: Got next segment. Inline: false Dec 30 09:44:57.623147022 (594h15m57.494230067s) [upload=520946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:44:57.623173563 (594h15m57.494256628s) [upload=520946]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:44:57.871673551 (594h15m57.742756596s) [seg_pos={0 0}, upload=520946]: Begin upload segment... Dec 30 09:44:57.871800984 (594h15m57.742884019s) [noise=true , address=45.91.116.9:4261 , piece_id=RWRQJOBRBFVPSJJSO5GEE5HGAUC5YOMNGSN7NK4PYRRP2ID34ZDA, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.871939869 (594h15m57.743022914s) [seg_pos={0 0}, upload=520946]: Done begin upload segment. Dec 30 09:44:57.872155219 (594h15m57.743238264s) [noise=true , address=45.91.116.12:4225 , piece_id=3ALOT7M65USCE2SPDMMYMUH7ZABAROLYQL2TXGSQNAIIVSVO266Q, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.874133145 (594h15m57.74521619s) [noise=true , address=145.79.227.19:4079 , piece_id=EWSYXMKBNBLJDOKGT5AIK5B3PX6Q7DGDEUGKU6B3KOCACO5YQZWQ, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.874996529 (594h15m57.746079605s) [noise=true , address=145.79.227.3:4149 , piece_id=4OMMF2SOZRL2X3FQEPG36A4TDZ5Y6ML7E2VU3QZLGO6BIZTUX5JQ, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.877030292 (594h15m57.748113327s) [noise=true , address=145.79.227.51:4410 , piece_id=SUXJYQ6LTZXHUIWPICDSZW2DWHQLBPSPACTX2566MZZUF32NT2KQ, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.877499646 (594h15m57.748582681s) [noise=true , address=145.79.227.50:4374 , piece_id=NFJ7DGXIBSCWHYGWBA6V7IHAGUY3JLITESB3ROVDYGNXH2APH7LA, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.881845311 (594h15m57.752928356s) [noise=true , address=145.79.227.25:4617 , piece_id=HDSEJJ5PWH24AH7SVTU33SWPFH5ETXUFCQVMSMIZDSXOVHQJL53Q, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.884276821 (594h15m57.755359856s) [noise=true , address=145.79.227.33:4870 , piece_id=WO4MUHFRHDNZQFBN7NP7E77KGL3XQ222MVNXV2JZZUI2ZSYZEV3Q, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.884588313 (594h15m57.755671348s) [noise=true , address=145.79.227.15:4874 , piece_id=EPT5WTJK4UDSWXPAYTROUQTJUCOS4V2AUXRIOXE27E3Y3D2ZOLTA, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.885959605 (594h15m57.75704264s) [noise=true , address=145.79.227.41:4087 , piece_id=C5PQ7VHX3GGDGGVT4QLBZZ5AT2IW2FJ4PEEGAAYHSQKKHREW7KYQ, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.887080730 (594h15m57.758163775s) [noise=true , address=145.79.227.52:4447 , piece_id=O6QCURQEPHEHW45KK3PV7FT573BWDBZL6Y5WMOEZXKEXFMTIWMGQ, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.888486356 (594h15m57.759569401s) [noise=true , address=77.90.31.15:4700 , piece_id=KAO3KG4G6DTVOA5WUSZ5FDH4ZPF57UPPMBZRLMQX5J6IIMNXDR5A, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.889751465 (594h15m57.760834501s) [noise=true , address=145.79.227.47:4267 , piece_id=NNMMR7KSQTKFQSMY2A6XWZVNYV2WEVH7MBYEEZHP3YLEDEMIYUWQ, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.890821583 (594h15m57.761904618s) [noise=true , address=145.79.227.31:4799 , piece_id=HGFAUEFCHYUT652V3SVXIRK4HFDTZQ7PRLU3DTMLSWXW6DJ5KHCQ, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.892028371 (594h15m57.763111406s) [noise=true , address=77.90.31.14:4663 , piece_id=4YENLUEZB52PFVDBHPOX2EHLJ74IRI42KYEVZPHBQMODUYBUQEJA, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:57.894879750 (594h15m57.765962785s) [noise=true , address=145.79.227.2:4115 , piece_id=DRVLXSM4QVGYXU7GDRV3JFVWGJCSHIJ4DPOELGJOLMMCZ7FET33Q, seg_pos={0 0}, upload=520946]: Uploading piece... Dec 30 09:44:59.588369427 (594h15m59.459452473s) [upload=520946]: Next returned nil segment Dec 30 09:44:59.588399084 (594h15m59.459482119s) [upload=520946]: Waiting for error group managing segments... Dec 30 09:44:59.869084860 (594h15m59.740167895s) [noise=true , address=145.79.227.50:4374 , piece_id=NFJ7DGXIBSCWHYGWBA6V7IHAGUY3JLITESB3ROVDYGNXH2APH7LA, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.877499113 (594h15m59.748582148s) [noise=true , address=145.79.227.3:4149 , piece_id=4OMMF2SOZRL2X3FQEPG36A4TDZ5Y6ML7E2VU3QZLGO6BIZTUX5JQ, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.878361345 (594h15m59.74944438s) [noise=true , address=145.79.227.25:4617 , piece_id=HDSEJJ5PWH24AH7SVTU33SWPFH5ETXUFCQVMSMIZDSXOVHQJL53Q, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.896653025 (594h15m59.76773606s) [noise=true , address=145.79.227.2:4115 , piece_id=DRVLXSM4QVGYXU7GDRV3JFVWGJCSHIJ4DPOELGJOLMMCZ7FET33Q, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.897270090 (594h15m59.768353165s) [noise=true , address=145.79.227.52:4447 , piece_id=O6QCURQEPHEHW45KK3PV7FT573BWDBZL6Y5WMOEZXKEXFMTIWMGQ, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.908624811 (594h15m59.779707856s) [noise=true , address=145.79.227.41:4087 , piece_id=C5PQ7VHX3GGDGGVT4QLBZZ5AT2IW2FJ4PEEGAAYHSQKKHREW7KYQ, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.920493381 (594h15m59.791576426s) [noise=true , address=145.79.227.31:4799 , piece_id=HGFAUEFCHYUT652V3SVXIRK4HFDTZQ7PRLU3DTMLSWXW6DJ5KHCQ, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.933222740 (594h15m59.804305775s) [noise=true , address=145.79.227.47:4267 , piece_id=NNMMR7KSQTKFQSMY2A6XWZVNYV2WEVH7MBYEEZHP3YLEDEMIYUWQ, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.938872648 (594h15m59.809955683s) [noise=true , address=145.79.227.19:4079 , piece_id=EWSYXMKBNBLJDOKGT5AIK5B3PX6Q7DGDEUGKU6B3KOCACO5YQZWQ, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.961010433 (594h15m59.832093478s) [noise=true , address=145.79.227.51:4410 , piece_id=SUXJYQ6LTZXHUIWPICDSZW2DWHQLBPSPACTX2566MZZUF32NT2KQ, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:44:59.980899836 (594h15m59.851982871s) [noise=true , address=145.79.227.15:4874 , piece_id=EPT5WTJK4UDSWXPAYTROUQTJUCOS4V2AUXRIOXE27E3Y3D2ZOLTA, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:45:00.183953407 (594h16m0.055036442s) [noise=true , address=77.90.31.14:4663 , piece_id=4YENLUEZB52PFVDBHPOX2EHLJ74IRI42KYEVZPHBQMODUYBUQEJA, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:45:00.309021879 (594h16m0.180104914s) [noise=true , address=45.91.116.12:4225 , piece_id=3ALOT7M65USCE2SPDMMYMUH7ZABAROLYQL2TXGSQNAIIVSVO266Q, seg_pos={0 0}, upload=520946]: Done uploading piece. err: <nil> Dec 30 09:45:00.309041327 (594h16m0.180124362s) [seg_pos={0 0}, upload=520946]: Segment reached optimal threshold of 13 pieces. Dec 30 09:45:00.309267126 (594h16m0.180350161s) [noise=true , address=45.91.116.9:4261 , piece_id=RWRQJOBRBFVPSJJSO5GEE5HGAUC5YOMNGSN7NK4PYRRP2ID34ZDA, seg_pos={0 0}, upload=520946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 30 09:45:00.309274149 (594h16m0.180357184s) [noise=true , address=45.91.116.9:4261 , piece_id=RWRQJOBRBFVPSJJSO5GEE5HGAUC5YOMNGSN7NK4PYRRP2ID34ZDA, seg_pos={0 0}, upload=520946]: Long tail cancellation due to optimal threshold Dec 30 09:45:00.309448141 (594h16m0.180531176s) [noise=true , address=77.90.31.15:4700 , piece_id=KAO3KG4G6DTVOA5WUSZ5FDH4ZPF57UPPMBZRLMQX5J6IIMNXDR5A, seg_pos={0 0}, upload=520946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 30 09:45:00.309469021 (594h16m0.180552056s) [noise=true , address=77.90.31.15:4700 , piece_id=KAO3KG4G6DTVOA5WUSZ5FDH4ZPF57UPPMBZRLMQX5J6IIMNXDR5A, seg_pos={0 0}, upload=520946]: Long tail cancellation due to optimal threshold Dec 30 09:45:00.309503036 (594h16m0.180586071s) [noise=true , address=145.79.227.33:4870 , piece_id=WO4MUHFRHDNZQFBN7NP7E77KGL3XQ222MVNXV2JZZUI2ZSYZEV3Q, seg_pos={0 0}, upload=520946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 09:45:00.309509938 (594h16m0.180592984s) [noise=true , address=145.79.227.33:4870 , piece_id=WO4MUHFRHDNZQFBN7NP7E77KGL3XQ222MVNXV2JZZUI2ZSYZEV3Q, seg_pos={0 0}, upload=520946]: Long tail cancellation due to optimal threshold Dec 30 09:45:00.309553351 (594h16m0.180636396s) [seg_pos={0 0}, upload=520946]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:45:00.309627412 (594h16m0.180710447s) [upload=520946]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:45:00.309633715 (594h16m0.18071676s) [upload=520946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:45:00.607168544 (594h16m0.478251589s) [upload=520946]: Done uploading segments... Dec 30 09:45:00.607191698 (594h16m0.478274743s) [upload=520946]: Upload finished. err: <nil> Duration: 30.002231654s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [...X.X...X........X.....X....XXX..X.X.X.......XX....X..X.XX.] duration: 3.528573757s trace: https://jaeger.dev.storj.io/trace/2a9d13babc34959 FAIL (2025-12-30 09:45:02 UTC) orbiter warning: context deadline exceeded Duration: 3.528607912s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...X..XX..XX......X...X...X...X...X..............X....X.X..] duration: 6.067541114s trace: https://jaeger.dev.storj.io/trace/2fd1e43ff8d3c231 FAIL (2025-12-30 09:44:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: DYQAZUPLD4SHYL3PETS7J6HQ2K57Q5YTX7NZ46BT6PQ2Z3Y6DQSQ) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: W2WBXQOJTYYONIG5THCAYNMSNF2LVMTP32LXSQJ4HSI26CVZXEVQ) context deadline exceeded ======LOGS====== Dec 30 09:42:41.420681463 (594h13m41.291764498s) [upload=520918]: Starting upload Dec 30 09:42:41.420748430 (594h13m41.291831465s) [upload=520918]: Done starting upload Dec 30 09:42:41.420871254 (594h13m41.291954289s) [upload=520918]: Uploading segments... Dec 30 09:42:41.421097455 (594h13m41.29218049s) [upload=520918]: Got next segment. Inline: false Dec 30 09:42:41.421139996 (594h13m41.292223041s) [upload=520918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:42:41.421145035 (594h13m41.29222807s) [upload=520918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:42:41.664290837 (594h13m41.535373882s) [seg_pos={0 0}, upload=520918]: Begin upload segment... Dec 30 09:42:41.664352755 (594h13m41.5354358s) [seg_pos={0 0}, upload=520918]: Done begin upload segment. Dec 30 09:42:41.664384626 (594h13m41.535467671s) [noise=true , address=145.79.227.4:4187 , piece_id=IUPK7XX2MEBWHJSTRGEH2WDJLTDYCF4OQTVJIW46J4NT2ZOUBW3Q, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.664397710 (594h13m41.535480745s) [noise=true , address=45.91.116.14:4191 , piece_id=3L5GF6ZLIKKAWNTTHV2ZLI3EEYZDSYASAUENXPL5RQB3A5W7FMQA, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.664844241 (594h13m41.535927276s) [noise=true , address=145.79.227.52:4445 , piece_id=SVLQSZKUQNVMREYXVR2GYW5RBN6RUT4AVHMI2Q5XTMUDYKEXKDKA, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.665108865 (594h13m41.53619191s) [noise=true , address=145.79.227.33:4871 , piece_id=UFYYIALH7PD4Y7ZACXSLNVPQKQDE56Q5CWNQRRMRL4JVMUHZVY6Q, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.665416882 (594h13m41.536499927s) [noise=true , address=77.90.31.18:4808 , piece_id=W2WBXQOJTYYONIG5THCAYNMSNF2LVMTP32LXSQJ4HSI26CVZXEVQ, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.665499889 (594h13m41.536582934s) [noise=true , address=145.79.227.26:4653 , piece_id=QBVOYX4JABG73AR36DN5Q44SNSLV7VGVTVEK6MD6SOMSFB2EXJ5A, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.665937683 (594h13m41.537020729s) [noise=true , address=145.79.227.6:4257 , piece_id=UPN327WC6TGMINBIUREF2M3WEND2TNSUZ7LNLVRH5LQFRXQEKZ7A, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.665953183 (594h13m41.537036228s) [noise=true , address=45.91.116.8:4048 , piece_id=JDO7OA2Q7VAO3H47Q37OGGVMPDPVYUH74MF6L3GNARSRPSGDHEXQ, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.666353976 (594h13m41.537437021s) [noise=true , address=145.79.227.20:4439 , piece_id=2UNMQN6PCJGFA3ZGBYXFXBZCCOWIUJVOQNOHXY5ZIT45UC6JI44A, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.666447003 (594h13m41.537530148s) [noise=true , address=145.79.227.13:4802 , piece_id=MIED4PFEU56DKYQFP2KRY6HLAORKQAC5RPHIIYTDVDVI2TG6LZCA, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.666958517 (594h13m41.538041552s) [noise=true , address=45.91.116.12:4227 , piece_id=P3P4APER4CHIZJYYGXC6P243UZFBVR4RHRHEHC6CBVOVGXTOCW5A, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.667312781 (594h13m41.538395816s) [noise=true , address=77.90.31.15:4700 , piece_id=DLHQG25YVHR3NUD4FTH25BNIW3GD3AZX4QLDV6KCG5FJRO4BWWCA, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.668292166 (594h13m41.539375201s) [noise=true , address=45.91.116.10:4084 , piece_id=DYQAZUPLD4SHYL3PETS7J6HQ2K57Q5YTX7NZ46BT6PQ2Z3Y6DQSQ, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.668729870 (594h13m41.539812915s) [noise=true , address=145.79.227.15:4876 , piece_id=NVEG5PQ663TJLLS5FBSWTABCI5YLNQNIQ63ZSNUURLW55DXQKPNQ, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.669099884 (594h13m41.540182929s) [noise=true , address=77.90.31.14:4664 , piece_id=O6KF7V6UNFEDBASGV7XZQQ3YNM7CRQAKLJSFKFAIL7OOLEZ5L43A, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.669458518 (594h13m41.540541553s) [noise=true , address=145.79.227.50:4374 , piece_id=SPVVAMA5BAUZBJFYQQDTWLILUO4XGJPDCCIZEQASIZCXKMWOFYBQ, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:43.368491761 (594h13m43.239574797s) [upload=520918]: Next returned nil segment Dec 30 09:42:43.368501199 (594h13m43.239584244s) [upload=520918]: Waiting for error group managing segments... Dec 30 09:42:43.647592360 (594h13m43.518675395s) [noise=true , address=145.79.227.4:4187 , piece_id=IUPK7XX2MEBWHJSTRGEH2WDJLTDYCF4OQTVJIW46J4NT2ZOUBW3Q, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.651569352 (594h13m43.522652397s) [noise=true , address=145.79.227.13:4802 , piece_id=MIED4PFEU56DKYQFP2KRY6HLAORKQAC5RPHIIYTDVDVI2TG6LZCA, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.654314209 (594h13m43.525397254s) [noise=true , address=145.79.227.15:4876 , piece_id=NVEG5PQ663TJLLS5FBSWTABCI5YLNQNIQ63ZSNUURLW55DXQKPNQ, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.657547506 (594h13m43.528630551s) [noise=true , address=145.79.227.50:4374 , piece_id=SPVVAMA5BAUZBJFYQQDTWLILUO4XGJPDCCIZEQASIZCXKMWOFYBQ, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.660691612 (594h13m43.531774658s) [noise=true , address=145.79.227.26:4653 , piece_id=QBVOYX4JABG73AR36DN5Q44SNSLV7VGVTVEK6MD6SOMSFB2EXJ5A, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.661970588 (594h13m43.533053623s) [noise=true , address=145.79.227.20:4439 , piece_id=2UNMQN6PCJGFA3ZGBYXFXBZCCOWIUJVOQNOHXY5ZIT45UC6JI44A, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.667982355 (594h13m43.53906539s) [noise=true , address=145.79.227.6:4257 , piece_id=UPN327WC6TGMINBIUREF2M3WEND2TNSUZ7LNLVRH5LQFRXQEKZ7A, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.668650678 (594h13m43.539733713s) [noise=true , address=145.79.227.52:4445 , piece_id=SVLQSZKUQNVMREYXVR2GYW5RBN6RUT4AVHMI2Q5XTMUDYKEXKDKA, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.932811597 (594h13m43.803894632s) [noise=true , address=77.90.31.15:4700 , piece_id=DLHQG25YVHR3NUD4FTH25BNIW3GD3AZX4QLDV6KCG5FJRO4BWWCA, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:44.057167974 (594h13m43.928251009s) [noise=true , address=45.91.116.12:4227 , piece_id=P3P4APER4CHIZJYYGXC6P243UZFBVR4RHRHEHC6CBVOVGXTOCW5A, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:44.084270132 (594h13m43.955353177s) [noise=true , address=45.91.116.10:4084 , piece_id=DYQAZUPLD4SHYL3PETS7J6HQ2K57Q5YTX7NZ46BT6PQ2Z3Y6DQSQ, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:44.568639708 (594h13m44.439722753s) [noise=true , address=77.90.31.18:4808 , piece_id=W2WBXQOJTYYONIG5THCAYNMSNF2LVMTP32LXSQJ4HSI26CVZXEVQ, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:44.700982713 (594h13m44.572065758s) [noise=true , address=45.91.116.14:4191 , piece_id=3L5GF6ZLIKKAWNTTHV2ZLI3EEYZDSYASAUENXPL5RQB3A5W7FMQA, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:44.701005837 (594h13m44.572088873s) [seg_pos={0 0}, upload=520918]: Segment reached optimal threshold of 13 pieces. Dec 30 09:42:44.701253589 (594h13m44.572336634s) [noise=true , address=145.79.227.33:4871 , piece_id=UFYYIALH7PD4Y7ZACXSLNVPQKQDE56Q5CWNQRRMRL4JVMUHZVY6Q, seg_pos={0 0}, upload=520918]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 30 09:42:44.701263367 (594h13m44.572346403s) [noise=true , address=145.79.227.33:4871 , piece_id=UFYYIALH7PD4Y7ZACXSLNVPQKQDE56Q5CWNQRRMRL4JVMUHZVY6Q, seg_pos={0 0}, upload=520918]: Long tail cancellation due to optimal threshold Dec 30 09:42:44.701355964 (594h13m44.572439009s) [noise=true , address=45.91.116.8:4048 , piece_id=JDO7OA2Q7VAO3H47Q37OGGVMPDPVYUH74MF6L3GNARSRPSGDHEXQ, seg_pos={0 0}, upload=520918]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 09:42:44.701365332 (594h13m44.572448377s) [noise=true , address=45.91.116.8:4048 , piece_id=JDO7OA2Q7VAO3H47Q37OGGVMPDPVYUH74MF6L3GNARSRPSGDHEXQ, seg_pos={0 0}, upload=520918]: Long tail cancellation due to optimal threshold Dec 30 09:42:44.701614416 (594h13m44.572697471s) [noise=true , address=77.90.31.14:4664 , piece_id=O6KF7V6UNFEDBASGV7XZQQ3YNM7CRQAKLJSFKFAIL7OOLEZ5L43A, seg_pos={0 0}, upload=520918]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: context canceled; context canceled Dec 30 09:42:44.701623523 (594h13m44.572706568s) [noise=true , address=77.90.31.14:4664 , piece_id=O6KF7V6UNFEDBASGV7XZQQ3YNM7CRQAKLJSFKFAIL7OOLEZ5L43A, seg_pos={0 0}, upload=520918]: Long tail cancellation due to optimal threshold Dec 30 09:42:44.701646597 (594h13m44.572729642s) [seg_pos={0 0}, upload=520918]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:42:44.701729114 (594h13m44.572812159s) [upload=520918]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:42:44.701736137 (594h13m44.572819182s) [upload=520918]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:42:44.981857018 (594h13m44.852940083s) [upload=520918]: Done uploading segments... Dec 30 09:42:44.981882056 (594h13m44.852965091s) [upload=520918]: Upload finished. err: <nil> Duration: 6.067572142s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [....X.X...X........X.....X....XXX..X.X.X.......XX....X..X.XX] duration: 5.001030098s trace: https://jaeger.dev.storj.io/trace/2a9d13babc34959 FAIL (2025-12-30 09:44:17 UTC) orbiter warning: context deadline exceeded Duration: 5.001053431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 27.325739561s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:44:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 27.325761754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 28.528526764s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:43:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 28.528542905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [X....X.X...X........X.....X....XXX..X.X.X.......XX....X..X.X] duration: 5.000825678s trace: https://jaeger.dev.storj.io/trace/75310942578d4aa8 FAIL (2025-12-30 09:43:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000846026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X...X..XX..XX......X...X...X...X...X..............X....X.X.] duration: 6.015083566s trace: https://jaeger.dev.storj.io/trace/2fd1e43ff8d3c231 FAIL (2025-12-30 09:43:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: DYQAZUPLD4SHYL3PETS7J6HQ2K57Q5YTX7NZ46BT6PQ2Z3Y6DQSQ) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: W2WBXQOJTYYONIG5THCAYNMSNF2LVMTP32LXSQJ4HSI26CVZXEVQ) context deadline exceeded ======LOGS====== Dec 30 09:42:41.420681463 (594h13m41.291764498s) [upload=520918]: Starting upload Dec 30 09:42:41.420748430 (594h13m41.291831465s) [upload=520918]: Done starting upload Dec 30 09:42:41.420871254 (594h13m41.291954289s) [upload=520918]: Uploading segments... Dec 30 09:42:41.421097455 (594h13m41.29218049s) [upload=520918]: Got next segment. Inline: false Dec 30 09:42:41.421139996 (594h13m41.292223041s) [upload=520918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:42:41.421145035 (594h13m41.29222807s) [upload=520918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:42:41.664290837 (594h13m41.535373882s) [seg_pos={0 0}, upload=520918]: Begin upload segment... Dec 30 09:42:41.664352755 (594h13m41.5354358s) [seg_pos={0 0}, upload=520918]: Done begin upload segment. Dec 30 09:42:41.664384626 (594h13m41.535467671s) [noise=true , address=145.79.227.4:4187 , piece_id=IUPK7XX2MEBWHJSTRGEH2WDJLTDYCF4OQTVJIW46J4NT2ZOUBW3Q, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.664397710 (594h13m41.535480745s) [noise=true , address=45.91.116.14:4191 , piece_id=3L5GF6ZLIKKAWNTTHV2ZLI3EEYZDSYASAUENXPL5RQB3A5W7FMQA, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.664844241 (594h13m41.535927276s) [noise=true , address=145.79.227.52:4445 , piece_id=SVLQSZKUQNVMREYXVR2GYW5RBN6RUT4AVHMI2Q5XTMUDYKEXKDKA, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.665108865 (594h13m41.53619191s) [noise=true , address=145.79.227.33:4871 , piece_id=UFYYIALH7PD4Y7ZACXSLNVPQKQDE56Q5CWNQRRMRL4JVMUHZVY6Q, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.665416882 (594h13m41.536499927s) [noise=true , address=77.90.31.18:4808 , piece_id=W2WBXQOJTYYONIG5THCAYNMSNF2LVMTP32LXSQJ4HSI26CVZXEVQ, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.665499889 (594h13m41.536582934s) [noise=true , address=145.79.227.26:4653 , piece_id=QBVOYX4JABG73AR36DN5Q44SNSLV7VGVTVEK6MD6SOMSFB2EXJ5A, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.665937683 (594h13m41.537020729s) [noise=true , address=145.79.227.6:4257 , piece_id=UPN327WC6TGMINBIUREF2M3WEND2TNSUZ7LNLVRH5LQFRXQEKZ7A, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.665953183 (594h13m41.537036228s) [noise=true , address=45.91.116.8:4048 , piece_id=JDO7OA2Q7VAO3H47Q37OGGVMPDPVYUH74MF6L3GNARSRPSGDHEXQ, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.666353976 (594h13m41.537437021s) [noise=true , address=145.79.227.20:4439 , piece_id=2UNMQN6PCJGFA3ZGBYXFXBZCCOWIUJVOQNOHXY5ZIT45UC6JI44A, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.666447003 (594h13m41.537530148s) [noise=true , address=145.79.227.13:4802 , piece_id=MIED4PFEU56DKYQFP2KRY6HLAORKQAC5RPHIIYTDVDVI2TG6LZCA, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.666958517 (594h13m41.538041552s) [noise=true , address=45.91.116.12:4227 , piece_id=P3P4APER4CHIZJYYGXC6P243UZFBVR4RHRHEHC6CBVOVGXTOCW5A, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.667312781 (594h13m41.538395816s) [noise=true , address=77.90.31.15:4700 , piece_id=DLHQG25YVHR3NUD4FTH25BNIW3GD3AZX4QLDV6KCG5FJRO4BWWCA, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.668292166 (594h13m41.539375201s) [noise=true , address=45.91.116.10:4084 , piece_id=DYQAZUPLD4SHYL3PETS7J6HQ2K57Q5YTX7NZ46BT6PQ2Z3Y6DQSQ, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.668729870 (594h13m41.539812915s) [noise=true , address=145.79.227.15:4876 , piece_id=NVEG5PQ663TJLLS5FBSWTABCI5YLNQNIQ63ZSNUURLW55DXQKPNQ, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.669099884 (594h13m41.540182929s) [noise=true , address=77.90.31.14:4664 , piece_id=O6KF7V6UNFEDBASGV7XZQQ3YNM7CRQAKLJSFKFAIL7OOLEZ5L43A, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.669458518 (594h13m41.540541553s) [noise=true , address=145.79.227.50:4374 , piece_id=SPVVAMA5BAUZBJFYQQDTWLILUO4XGJPDCCIZEQASIZCXKMWOFYBQ, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:43.368491761 (594h13m43.239574797s) [upload=520918]: Next returned nil segment Dec 30 09:42:43.368501199 (594h13m43.239584244s) [upload=520918]: Waiting for error group managing segments... Dec 30 09:42:43.647592360 (594h13m43.518675395s) [noise=true , address=145.79.227.4:4187 , piece_id=IUPK7XX2MEBWHJSTRGEH2WDJLTDYCF4OQTVJIW46J4NT2ZOUBW3Q, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.651569352 (594h13m43.522652397s) [noise=true , address=145.79.227.13:4802 , piece_id=MIED4PFEU56DKYQFP2KRY6HLAORKQAC5RPHIIYTDVDVI2TG6LZCA, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.654314209 (594h13m43.525397254s) [noise=true , address=145.79.227.15:4876 , piece_id=NVEG5PQ663TJLLS5FBSWTABCI5YLNQNIQ63ZSNUURLW55DXQKPNQ, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.657547506 (594h13m43.528630551s) [noise=true , address=145.79.227.50:4374 , piece_id=SPVVAMA5BAUZBJFYQQDTWLILUO4XGJPDCCIZEQASIZCXKMWOFYBQ, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.660691612 (594h13m43.531774658s) [noise=true , address=145.79.227.26:4653 , piece_id=QBVOYX4JABG73AR36DN5Q44SNSLV7VGVTVEK6MD6SOMSFB2EXJ5A, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.661970588 (594h13m43.533053623s) [noise=true , address=145.79.227.20:4439 , piece_id=2UNMQN6PCJGFA3ZGBYXFXBZCCOWIUJVOQNOHXY5ZIT45UC6JI44A, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.667982355 (594h13m43.53906539s) [noise=true , address=145.79.227.6:4257 , piece_id=UPN327WC6TGMINBIUREF2M3WEND2TNSUZ7LNLVRH5LQFRXQEKZ7A, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.668650678 (594h13m43.539733713s) [noise=true , address=145.79.227.52:4445 , piece_id=SVLQSZKUQNVMREYXVR2GYW5RBN6RUT4AVHMI2Q5XTMUDYKEXKDKA, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.932811597 (594h13m43.803894632s) [noise=true , address=77.90.31.15:4700 , piece_id=DLHQG25YVHR3NUD4FTH25BNIW3GD3AZX4QLDV6KCG5FJRO4BWWCA, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:44.057167974 (594h13m43.928251009s) [noise=true , address=45.91.116.12:4227 , piece_id=P3P4APER4CHIZJYYGXC6P243UZFBVR4RHRHEHC6CBVOVGXTOCW5A, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:44.084270132 (594h13m43.955353177s) [noise=true , address=45.91.116.10:4084 , piece_id=DYQAZUPLD4SHYL3PETS7J6HQ2K57Q5YTX7NZ46BT6PQ2Z3Y6DQSQ, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:44.568639708 (594h13m44.439722753s) [noise=true , address=77.90.31.18:4808 , piece_id=W2WBXQOJTYYONIG5THCAYNMSNF2LVMTP32LXSQJ4HSI26CVZXEVQ, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:44.700982713 (594h13m44.572065758s) [noise=true , address=45.91.116.14:4191 , piece_id=3L5GF6ZLIKKAWNTTHV2ZLI3EEYZDSYASAUENXPL5RQB3A5W7FMQA, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:44.701005837 (594h13m44.572088873s) [seg_pos={0 0}, upload=520918]: Segment reached optimal threshold of 13 pieces. Dec 30 09:42:44.701253589 (594h13m44.572336634s) [noise=true , address=145.79.227.33:4871 , piece_id=UFYYIALH7PD4Y7ZACXSLNVPQKQDE56Q5CWNQRRMRL4JVMUHZVY6Q, seg_pos={0 0}, upload=520918]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 30 09:42:44.701263367 (594h13m44.572346403s) [noise=true , address=145.79.227.33:4871 , piece_id=UFYYIALH7PD4Y7ZACXSLNVPQKQDE56Q5CWNQRRMRL4JVMUHZVY6Q, seg_pos={0 0}, upload=520918]: Long tail cancellation due to optimal threshold Dec 30 09:42:44.701355964 (594h13m44.572439009s) [noise=true , address=45.91.116.8:4048 , piece_id=JDO7OA2Q7VAO3H47Q37OGGVMPDPVYUH74MF6L3GNARSRPSGDHEXQ, seg_pos={0 0}, upload=520918]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 09:42:44.701365332 (594h13m44.572448377s) [noise=true , address=45.91.116.8:4048 , piece_id=JDO7OA2Q7VAO3H47Q37OGGVMPDPVYUH74MF6L3GNARSRPSGDHEXQ, seg_pos={0 0}, upload=520918]: Long tail cancellation due to optimal threshold Dec 30 09:42:44.701614416 (594h13m44.572697471s) [noise=true , address=77.90.31.14:4664 , piece_id=O6KF7V6UNFEDBASGV7XZQQ3YNM7CRQAKLJSFKFAIL7OOLEZ5L43A, seg_pos={0 0}, upload=520918]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: context canceled; context canceled Dec 30 09:42:44.701623523 (594h13m44.572706568s) [noise=true , address=77.90.31.14:4664 , piece_id=O6KF7V6UNFEDBASGV7XZQQ3YNM7CRQAKLJSFKFAIL7OOLEZ5L43A, seg_pos={0 0}, upload=520918]: Long tail cancellation due to optimal threshold Dec 30 09:42:44.701646597 (594h13m44.572729642s) [seg_pos={0 0}, upload=520918]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:42:44.701729114 (594h13m44.572812159s) [upload=520918]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:42:44.701736137 (594h13m44.572819182s) [upload=520918]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:42:44.981857018 (594h13m44.852940083s) [upload=520918]: Done uploading segments... Dec 30 09:42:44.981882056 (594h13m44.852965091s) [upload=520918]: Upload finished. err: <nil> Duration: 6.015113413s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 33.755097383s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:43:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 33.755124865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X...X..XX..XX......X...X...X...X...X..............X....X.X] duration: 30.003005661s trace: https://jaeger.dev.storj.io/trace/2fd1e43ff8d3c231 FAIL (2025-12-30 09:43:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: DYQAZUPLD4SHYL3PETS7J6HQ2K57Q5YTX7NZ46BT6PQ2Z3Y6DQSQ) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: W2WBXQOJTYYONIG5THCAYNMSNF2LVMTP32LXSQJ4HSI26CVZXEVQ) context deadline exceeded ======LOGS====== Dec 30 09:42:41.420681463 (594h13m41.291764498s) [upload=520918]: Starting upload Dec 30 09:42:41.420748430 (594h13m41.291831465s) [upload=520918]: Done starting upload Dec 30 09:42:41.420871254 (594h13m41.291954289s) [upload=520918]: Uploading segments... Dec 30 09:42:41.421097455 (594h13m41.29218049s) [upload=520918]: Got next segment. Inline: false Dec 30 09:42:41.421139996 (594h13m41.292223041s) [upload=520918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:42:41.421145035 (594h13m41.29222807s) [upload=520918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:42:41.664290837 (594h13m41.535373882s) [seg_pos={0 0}, upload=520918]: Begin upload segment... Dec 30 09:42:41.664352755 (594h13m41.5354358s) [seg_pos={0 0}, upload=520918]: Done begin upload segment. Dec 30 09:42:41.664384626 (594h13m41.535467671s) [noise=true , address=145.79.227.4:4187 , piece_id=IUPK7XX2MEBWHJSTRGEH2WDJLTDYCF4OQTVJIW46J4NT2ZOUBW3Q, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.664397710 (594h13m41.535480745s) [noise=true , address=45.91.116.14:4191 , piece_id=3L5GF6ZLIKKAWNTTHV2ZLI3EEYZDSYASAUENXPL5RQB3A5W7FMQA, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.664844241 (594h13m41.535927276s) [noise=true , address=145.79.227.52:4445 , piece_id=SVLQSZKUQNVMREYXVR2GYW5RBN6RUT4AVHMI2Q5XTMUDYKEXKDKA, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.665108865 (594h13m41.53619191s) [noise=true , address=145.79.227.33:4871 , piece_id=UFYYIALH7PD4Y7ZACXSLNVPQKQDE56Q5CWNQRRMRL4JVMUHZVY6Q, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.665416882 (594h13m41.536499927s) [noise=true , address=77.90.31.18:4808 , piece_id=W2WBXQOJTYYONIG5THCAYNMSNF2LVMTP32LXSQJ4HSI26CVZXEVQ, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.665499889 (594h13m41.536582934s) [noise=true , address=145.79.227.26:4653 , piece_id=QBVOYX4JABG73AR36DN5Q44SNSLV7VGVTVEK6MD6SOMSFB2EXJ5A, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.665937683 (594h13m41.537020729s) [noise=true , address=145.79.227.6:4257 , piece_id=UPN327WC6TGMINBIUREF2M3WEND2TNSUZ7LNLVRH5LQFRXQEKZ7A, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.665953183 (594h13m41.537036228s) [noise=true , address=45.91.116.8:4048 , piece_id=JDO7OA2Q7VAO3H47Q37OGGVMPDPVYUH74MF6L3GNARSRPSGDHEXQ, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.666353976 (594h13m41.537437021s) [noise=true , address=145.79.227.20:4439 , piece_id=2UNMQN6PCJGFA3ZGBYXFXBZCCOWIUJVOQNOHXY5ZIT45UC6JI44A, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.666447003 (594h13m41.537530148s) [noise=true , address=145.79.227.13:4802 , piece_id=MIED4PFEU56DKYQFP2KRY6HLAORKQAC5RPHIIYTDVDVI2TG6LZCA, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.666958517 (594h13m41.538041552s) [noise=true , address=45.91.116.12:4227 , piece_id=P3P4APER4CHIZJYYGXC6P243UZFBVR4RHRHEHC6CBVOVGXTOCW5A, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.667312781 (594h13m41.538395816s) [noise=true , address=77.90.31.15:4700 , piece_id=DLHQG25YVHR3NUD4FTH25BNIW3GD3AZX4QLDV6KCG5FJRO4BWWCA, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.668292166 (594h13m41.539375201s) [noise=true , address=45.91.116.10:4084 , piece_id=DYQAZUPLD4SHYL3PETS7J6HQ2K57Q5YTX7NZ46BT6PQ2Z3Y6DQSQ, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.668729870 (594h13m41.539812915s) [noise=true , address=145.79.227.15:4876 , piece_id=NVEG5PQ663TJLLS5FBSWTABCI5YLNQNIQ63ZSNUURLW55DXQKPNQ, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.669099884 (594h13m41.540182929s) [noise=true , address=77.90.31.14:4664 , piece_id=O6KF7V6UNFEDBASGV7XZQQ3YNM7CRQAKLJSFKFAIL7OOLEZ5L43A, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:41.669458518 (594h13m41.540541553s) [noise=true , address=145.79.227.50:4374 , piece_id=SPVVAMA5BAUZBJFYQQDTWLILUO4XGJPDCCIZEQASIZCXKMWOFYBQ, seg_pos={0 0}, upload=520918]: Uploading piece... Dec 30 09:42:43.368491761 (594h13m43.239574797s) [upload=520918]: Next returned nil segment Dec 30 09:42:43.368501199 (594h13m43.239584244s) [upload=520918]: Waiting for error group managing segments... Dec 30 09:42:43.647592360 (594h13m43.518675395s) [noise=true , address=145.79.227.4:4187 , piece_id=IUPK7XX2MEBWHJSTRGEH2WDJLTDYCF4OQTVJIW46J4NT2ZOUBW3Q, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.651569352 (594h13m43.522652397s) [noise=true , address=145.79.227.13:4802 , piece_id=MIED4PFEU56DKYQFP2KRY6HLAORKQAC5RPHIIYTDVDVI2TG6LZCA, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.654314209 (594h13m43.525397254s) [noise=true , address=145.79.227.15:4876 , piece_id=NVEG5PQ663TJLLS5FBSWTABCI5YLNQNIQ63ZSNUURLW55DXQKPNQ, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.657547506 (594h13m43.528630551s) [noise=true , address=145.79.227.50:4374 , piece_id=SPVVAMA5BAUZBJFYQQDTWLILUO4XGJPDCCIZEQASIZCXKMWOFYBQ, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.660691612 (594h13m43.531774658s) [noise=true , address=145.79.227.26:4653 , piece_id=QBVOYX4JABG73AR36DN5Q44SNSLV7VGVTVEK6MD6SOMSFB2EXJ5A, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.661970588 (594h13m43.533053623s) [noise=true , address=145.79.227.20:4439 , piece_id=2UNMQN6PCJGFA3ZGBYXFXBZCCOWIUJVOQNOHXY5ZIT45UC6JI44A, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.667982355 (594h13m43.53906539s) [noise=true , address=145.79.227.6:4257 , piece_id=UPN327WC6TGMINBIUREF2M3WEND2TNSUZ7LNLVRH5LQFRXQEKZ7A, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.668650678 (594h13m43.539733713s) [noise=true , address=145.79.227.52:4445 , piece_id=SVLQSZKUQNVMREYXVR2GYW5RBN6RUT4AVHMI2Q5XTMUDYKEXKDKA, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:43.932811597 (594h13m43.803894632s) [noise=true , address=77.90.31.15:4700 , piece_id=DLHQG25YVHR3NUD4FTH25BNIW3GD3AZX4QLDV6KCG5FJRO4BWWCA, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:44.057167974 (594h13m43.928251009s) [noise=true , address=45.91.116.12:4227 , piece_id=P3P4APER4CHIZJYYGXC6P243UZFBVR4RHRHEHC6CBVOVGXTOCW5A, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:44.084270132 (594h13m43.955353177s) [noise=true , address=45.91.116.10:4084 , piece_id=DYQAZUPLD4SHYL3PETS7J6HQ2K57Q5YTX7NZ46BT6PQ2Z3Y6DQSQ, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:44.568639708 (594h13m44.439722753s) [noise=true , address=77.90.31.18:4808 , piece_id=W2WBXQOJTYYONIG5THCAYNMSNF2LVMTP32LXSQJ4HSI26CVZXEVQ, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:44.700982713 (594h13m44.572065758s) [noise=true , address=45.91.116.14:4191 , piece_id=3L5GF6ZLIKKAWNTTHV2ZLI3EEYZDSYASAUENXPL5RQB3A5W7FMQA, seg_pos={0 0}, upload=520918]: Done uploading piece. err: <nil> Dec 30 09:42:44.701005837 (594h13m44.572088873s) [seg_pos={0 0}, upload=520918]: Segment reached optimal threshold of 13 pieces. Dec 30 09:42:44.701253589 (594h13m44.572336634s) [noise=true , address=145.79.227.33:4871 , piece_id=UFYYIALH7PD4Y7ZACXSLNVPQKQDE56Q5CWNQRRMRL4JVMUHZVY6Q, seg_pos={0 0}, upload=520918]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 30 09:42:44.701263367 (594h13m44.572346403s) [noise=true , address=145.79.227.33:4871 , piece_id=UFYYIALH7PD4Y7ZACXSLNVPQKQDE56Q5CWNQRRMRL4JVMUHZVY6Q, seg_pos={0 0}, upload=520918]: Long tail cancellation due to optimal threshold Dec 30 09:42:44.701355964 (594h13m44.572439009s) [noise=true , address=45.91.116.8:4048 , piece_id=JDO7OA2Q7VAO3H47Q37OGGVMPDPVYUH74MF6L3GNARSRPSGDHEXQ, seg_pos={0 0}, upload=520918]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 09:42:44.701365332 (594h13m44.572448377s) [noise=true , address=45.91.116.8:4048 , piece_id=JDO7OA2Q7VAO3H47Q37OGGVMPDPVYUH74MF6L3GNARSRPSGDHEXQ, seg_pos={0 0}, upload=520918]: Long tail cancellation due to optimal threshold Dec 30 09:42:44.701614416 (594h13m44.572697471s) [noise=true , address=77.90.31.14:4664 , piece_id=O6KF7V6UNFEDBASGV7XZQQ3YNM7CRQAKLJSFKFAIL7OOLEZ5L43A, seg_pos={0 0}, upload=520918]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: context canceled; context canceled Dec 30 09:42:44.701623523 (594h13m44.572706568s) [noise=true , address=77.90.31.14:4664 , piece_id=O6KF7V6UNFEDBASGV7XZQQ3YNM7CRQAKLJSFKFAIL7OOLEZ5L43A, seg_pos={0 0}, upload=520918]: Long tail cancellation due to optimal threshold Dec 30 09:42:44.701646597 (594h13m44.572729642s) [seg_pos={0 0}, upload=520918]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:42:44.701729114 (594h13m44.572812159s) [upload=520918]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:42:44.701736137 (594h13m44.572819182s) [upload=520918]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:42:44.981857018 (594h13m44.852940083s) [upload=520918]: Done uploading segments... Dec 30 09:42:44.981882056 (594h13m44.852965091s) [upload=520918]: Upload finished. err: <nil> Duration: 30.003035017s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.X....X.X...X........X.....X....XXX..X.X.X.......XX....X..X.] duration: 5.001806966s trace: https://jaeger.dev.storj.io/trace/2f918a0f83ff1119 FAIL (2025-12-30 09:42:35 UTC) orbiter warning: context deadline exceeded Duration: 5.001830742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 35.94597274s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:42:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 35.946005904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X...X..XX..XX......X...X...X...X...X..............X....X.] duration: 5.901890185s trace: https://jaeger.dev.storj.io/trace/24544087911a0029 FAIL (2025-12-30 09:42:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: RDB5KHAM4ZNUWC6HB6FDRSIZUQRSHYBXBWB66OBICTFBA5TKPMSQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: UB64CJMQSTLO4Z6P4VNFPFD3SCQ5HY6AQ5EDZXLCH3VNPZZTQRRA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: XDJXNXEY5EUHYOUAWI2EZ5WFKT7QYYSDX6P436HGA6AKAJYH5S5Q) context deadline exceeded ======LOGS====== Dec 30 09:40:58.687347827 (594h11m58.558430872s) [upload=520892]: Starting upload Dec 30 09:40:58.687407080 (594h11m58.558490135s) [upload=520892]: Done starting upload Dec 30 09:40:58.687588064 (594h11m58.55867111s) [upload=520892]: Uploading segments... Dec 30 09:40:58.687803214 (594h11m58.558886249s) [upload=520892]: Got next segment. Inline: false Dec 30 09:40:58.687873037 (594h11m58.558956092s) [upload=520892]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:40:58.687879260 (594h11m58.558962315s) [upload=520892]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:40:58.930558004 (594h11m58.801641039s) [seg_pos={0 0}, upload=520892]: Begin upload segment... Dec 30 09:40:58.930603632 (594h11m58.801686677s) [seg_pos={0 0}, upload=520892]: Done begin upload segment. Dec 30 09:40:58.930634700 (594h11m58.801717735s) [noise=true , address=145.79.227.36:4518 , piece_id=V6D77VUW7P5N7D6PJTVFB3MMP7QBOGUAXFIBLXVD76FGCU5JJI4Q, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.930643648 (594h11m58.801726683s) [noise=true , address=145.79.227.25:4617 , piece_id=VRWMVGO656LO5KDP32ZPT65L7HR4YGA63HVL6BWWGIWT2HO2EM7A, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.931088164 (594h11m58.802171199s) [noise=true , address=145.79.227.40:4051 , piece_id=7YYY4NGG3DG4LJS2HJU5JK5VFDJPI7MCRFNBCMY6FLMQAVEDN7IA, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.931382274 (594h11m58.802465309s) [noise=true , address=77.90.31.21:4878 , piece_id=RDB5KHAM4ZNUWC6HB6FDRSIZUQRSHYBXBWB66OBICTFBA5TKPMSQ, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.931801913 (594h11m58.802884948s) [noise=true , address=77.90.31.11:4554 , piece_id=XJNMRVSZFFZPYGWQMPC6WBX7VHEWD7HNIQB6UXW3SUP3KNY3ZJQQ, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.932096394 (594h11m58.803179429s) [noise=true , address=145.79.227.52:4446 , piece_id=CFN2DUXQZLD4KJCU3Q3YNDBLJEC4UQNKKRU7TNYYEYDUIXBPKMTA, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.932398159 (594h11m58.803481194s) [noise=true , address=45.91.116.8:4045 , piece_id=775ZABQ6Y6DFYWNWSNIETMTV6XUKC2ZCA6ZOQRBH34YIORFPJCIQ, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.932694894 (594h11m58.803777929s) [noise=true , address=145.79.227.6:4257 , piece_id=BTV2GMJ6UP74BJZLMYMIVRHXB7SQHHKOMJUZ2TCC3NI7YFYMOCQQ, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.933042726 (594h11m58.804125771s) [noise=true , address=145.79.227.32:4835 , piece_id=QV6KUS4JCIKX6RVNXVKOVVYGNL5DDL4JGXZYXOEV4F3KFQA5UZGQ, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.934260255 (594h11m58.80534329s) [noise=true , address=145.79.227.43:4157 , piece_id=DMTHBEXA4KFU3VUY24GA3IZ4CGMGAPAPHGVXPFMAT3KW5KCRX3GA, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.937190254 (594h11m58.808273289s) [noise=true , address=145.79.227.4:4185 , piece_id=QAN24H5MFCTSR7JF2VONIGQMKLPV365O547UHN5OV3DPUQCWRB4Q, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.940608202 (594h11m58.811691237s) [noise=true , address=145.79.227.23:4545 , piece_id=KCXQ6LTKZH454Q5NWWCPXPYX2WJWOHIDBYVEQWNPURXTFM5GYNFQ, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.941468150 (594h11m58.812551195s) [noise=true , address=77.90.31.17:4773 , piece_id=UB64CJMQSTLO4Z6P4VNFPFD3SCQ5HY6AQ5EDZXLCH3VNPZZTQRRA, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.943827442 (594h11m58.814910477s) [noise=true , address=145.79.227.45:4231 , piece_id=I7XS3UODLOAJORZ7K6BWX5GCJQBMFHWOYWDARUSQ34FRKSAFBKCA, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.946225057 (594h11m58.817308092s) [noise=true , address=45.91.116.13:4155 , piece_id=XDJXNXEY5EUHYOUAWI2EZ5WFKT7QYYSDX6P436HGA6AKAJYH5S5Q, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.948897196 (594h11m58.819980231s) [noise=true , address=145.79.227.9:4366 , piece_id=IGZ7H6AJYRANISEGHFRRN77ZZC6WWGHJM7RL4L3CP57GWKAL7AXQ, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:41:00.630434513 (594h12m0.501517558s) [upload=520892]: Next returned nil segment Dec 30 09:41:00.630448059 (594h12m0.501531094s) [upload=520892]: Waiting for error group managing segments... Dec 30 09:41:00.910601240 (594h12m0.781684285s) [noise=true , address=145.79.227.6:4257 , piece_id=BTV2GMJ6UP74BJZLMYMIVRHXB7SQHHKOMJUZ2TCC3NI7YFYMOCQQ, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:00.920090881 (594h12m0.791173916s) [noise=true , address=145.79.227.32:4835 , piece_id=QV6KUS4JCIKX6RVNXVKOVVYGNL5DDL4JGXZYXOEV4F3KFQA5UZGQ, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:00.920803998 (594h12m0.791887033s) [noise=true , address=145.79.227.52:4446 , piece_id=CFN2DUXQZLD4KJCU3Q3YNDBLJEC4UQNKKRU7TNYYEYDUIXBPKMTA, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:00.921710415 (594h12m0.79279346s) [noise=true , address=145.79.227.36:4518 , piece_id=V6D77VUW7P5N7D6PJTVFB3MMP7QBOGUAXFIBLXVD76FGCU5JJI4Q, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:00.941792123 (594h12m0.812875168s) [noise=true , address=145.79.227.4:4185 , piece_id=QAN24H5MFCTSR7JF2VONIGQMKLPV365O547UHN5OV3DPUQCWRB4Q, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:00.945049606 (594h12m0.816132651s) [noise=true , address=145.79.227.25:4617 , piece_id=VRWMVGO656LO5KDP32ZPT65L7HR4YGA63HVL6BWWGIWT2HO2EM7A, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:00.956688779 (594h12m0.827771814s) [noise=true , address=145.79.227.23:4545 , piece_id=KCXQ6LTKZH454Q5NWWCPXPYX2WJWOHIDBYVEQWNPURXTFM5GYNFQ, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:01.023280592 (594h12m0.894363637s) [noise=true , address=145.79.227.45:4231 , piece_id=I7XS3UODLOAJORZ7K6BWX5GCJQBMFHWOYWDARUSQ34FRKSAFBKCA, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:01.033536321 (594h12m0.904619356s) [noise=true , address=145.79.227.9:4366 , piece_id=IGZ7H6AJYRANISEGHFRRN77ZZC6WWGHJM7RL4L3CP57GWKAL7AXQ, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:01.193569212 (594h12m1.064652257s) [noise=true , address=77.90.31.17:4773 , piece_id=UB64CJMQSTLO4Z6P4VNFPFD3SCQ5HY6AQ5EDZXLCH3VNPZZTQRRA, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:01.231586694 (594h12m1.102669739s) [noise=true , address=77.90.31.11:4554 , piece_id=XJNMRVSZFFZPYGWQMPC6WBX7VHEWD7HNIQB6UXW3SUP3KNY3ZJQQ, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:01.302711759 (594h12m1.173794804s) [noise=true , address=45.91.116.13:4155 , piece_id=XDJXNXEY5EUHYOUAWI2EZ5WFKT7QYYSDX6P436HGA6AKAJYH5S5Q, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:01.895766872 (594h12m1.766849907s) [noise=true , address=77.90.31.21:4878 , piece_id=RDB5KHAM4ZNUWC6HB6FDRSIZUQRSHYBXBWB66OBICTFBA5TKPMSQ, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:01.895795697 (594h12m1.766878742s) [seg_pos={0 0}, upload=520892]: Segment reached optimal threshold of 13 pieces. Dec 30 09:41:01.896047607 (594h12m1.767130662s) [noise=true , address=145.79.227.43:4157 , piece_id=DMTHBEXA4KFU3VUY24GA3IZ4CGMGAPAPHGVXPFMAT3KW5KCRX3GA, seg_pos={0 0}, upload=520892]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 09:41:01.896058228 (594h12m1.767141273s) [noise=true , address=145.79.227.43:4157 , piece_id=DMTHBEXA4KFU3VUY24GA3IZ4CGMGAPAPHGVXPFMAT3KW5KCRX3GA, seg_pos={0 0}, upload=520892]: Long tail cancellation due to optimal threshold Dec 30 09:41:01.896281753 (594h12m1.767364798s) [noise=true , address=45.91.116.8:4045 , piece_id=775ZABQ6Y6DFYWNWSNIETMTV6XUKC2ZCA6ZOQRBH34YIORFPJCIQ, seg_pos={0 0}, upload=520892]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 30 09:41:01.896300399 (594h12m1.767383444s) [noise=true , address=45.91.116.8:4045 , piece_id=775ZABQ6Y6DFYWNWSNIETMTV6XUKC2ZCA6ZOQRBH34YIORFPJCIQ, seg_pos={0 0}, upload=520892]: Long tail cancellation due to optimal threshold Dec 30 09:41:01.896464490 (594h12m1.767547536s) [noise=true , address=145.79.227.40:4051 , piece_id=7YYY4NGG3DG4LJS2HJU5JK5VFDJPI7MCRFNBCMY6FLMQAVEDN7IA, seg_pos={0 0}, upload=520892]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 09:41:01.896476674 (594h12m1.767559719s) [noise=true , address=145.79.227.40:4051 , piece_id=7YYY4NGG3DG4LJS2HJU5JK5VFDJPI7MCRFNBCMY6FLMQAVEDN7IA, seg_pos={0 0}, upload=520892]: Long tail cancellation due to optimal threshold Dec 30 09:41:01.896501381 (594h12m1.767584426s) [seg_pos={0 0}, upload=520892]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:41:01.896603906 (594h12m1.767686951s) [upload=520892]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:41:01.896612112 (594h12m1.767695157s) [upload=520892]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:41:02.186287346 (594h12m2.057370391s) [upload=520892]: Done uploading segments... Dec 30 09:41:02.186314738 (594h12m2.057397783s) [upload=520892]: Upload finished. err: <nil> Duration: 5.901915683s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [X.X....X.X...X........X.....X....XXX..X.X.X.......XX....X..X] duration: 5.001220419s trace: https://jaeger.dev.storj.io/trace/2f918a0f83ff1119 FAIL (2025-12-30 09:41:41 UTC) orbiter warning: context deadline exceeded Duration: 5.001246599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..........................X................................] duration: 44.992372435s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:41:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 44.992404886s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X...X..XX..XX......X...X...X...X...X..............X....X] duration: 30.002765066s trace: https://jaeger.dev.storj.io/trace/24544087911a0029 FAIL (2025-12-30 09:41:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: RDB5KHAM4ZNUWC6HB6FDRSIZUQRSHYBXBWB66OBICTFBA5TKPMSQ) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: UB64CJMQSTLO4Z6P4VNFPFD3SCQ5HY6AQ5EDZXLCH3VNPZZTQRRA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: XDJXNXEY5EUHYOUAWI2EZ5WFKT7QYYSDX6P436HGA6AKAJYH5S5Q) context deadline exceeded ======LOGS====== Dec 30 09:40:58.687347827 (594h11m58.558430872s) [upload=520892]: Starting upload Dec 30 09:40:58.687407080 (594h11m58.558490135s) [upload=520892]: Done starting upload Dec 30 09:40:58.687588064 (594h11m58.55867111s) [upload=520892]: Uploading segments... Dec 30 09:40:58.687803214 (594h11m58.558886249s) [upload=520892]: Got next segment. Inline: false Dec 30 09:40:58.687873037 (594h11m58.558956092s) [upload=520892]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:40:58.687879260 (594h11m58.558962315s) [upload=520892]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:40:58.930558004 (594h11m58.801641039s) [seg_pos={0 0}, upload=520892]: Begin upload segment... Dec 30 09:40:58.930603632 (594h11m58.801686677s) [seg_pos={0 0}, upload=520892]: Done begin upload segment. Dec 30 09:40:58.930634700 (594h11m58.801717735s) [noise=true , address=145.79.227.36:4518 , piece_id=V6D77VUW7P5N7D6PJTVFB3MMP7QBOGUAXFIBLXVD76FGCU5JJI4Q, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.930643648 (594h11m58.801726683s) [noise=true , address=145.79.227.25:4617 , piece_id=VRWMVGO656LO5KDP32ZPT65L7HR4YGA63HVL6BWWGIWT2HO2EM7A, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.931088164 (594h11m58.802171199s) [noise=true , address=145.79.227.40:4051 , piece_id=7YYY4NGG3DG4LJS2HJU5JK5VFDJPI7MCRFNBCMY6FLMQAVEDN7IA, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.931382274 (594h11m58.802465309s) [noise=true , address=77.90.31.21:4878 , piece_id=RDB5KHAM4ZNUWC6HB6FDRSIZUQRSHYBXBWB66OBICTFBA5TKPMSQ, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.931801913 (594h11m58.802884948s) [noise=true , address=77.90.31.11:4554 , piece_id=XJNMRVSZFFZPYGWQMPC6WBX7VHEWD7HNIQB6UXW3SUP3KNY3ZJQQ, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.932096394 (594h11m58.803179429s) [noise=true , address=145.79.227.52:4446 , piece_id=CFN2DUXQZLD4KJCU3Q3YNDBLJEC4UQNKKRU7TNYYEYDUIXBPKMTA, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.932398159 (594h11m58.803481194s) [noise=true , address=45.91.116.8:4045 , piece_id=775ZABQ6Y6DFYWNWSNIETMTV6XUKC2ZCA6ZOQRBH34YIORFPJCIQ, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.932694894 (594h11m58.803777929s) [noise=true , address=145.79.227.6:4257 , piece_id=BTV2GMJ6UP74BJZLMYMIVRHXB7SQHHKOMJUZ2TCC3NI7YFYMOCQQ, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.933042726 (594h11m58.804125771s) [noise=true , address=145.79.227.32:4835 , piece_id=QV6KUS4JCIKX6RVNXVKOVVYGNL5DDL4JGXZYXOEV4F3KFQA5UZGQ, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.934260255 (594h11m58.80534329s) [noise=true , address=145.79.227.43:4157 , piece_id=DMTHBEXA4KFU3VUY24GA3IZ4CGMGAPAPHGVXPFMAT3KW5KCRX3GA, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.937190254 (594h11m58.808273289s) [noise=true , address=145.79.227.4:4185 , piece_id=QAN24H5MFCTSR7JF2VONIGQMKLPV365O547UHN5OV3DPUQCWRB4Q, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.940608202 (594h11m58.811691237s) [noise=true , address=145.79.227.23:4545 , piece_id=KCXQ6LTKZH454Q5NWWCPXPYX2WJWOHIDBYVEQWNPURXTFM5GYNFQ, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.941468150 (594h11m58.812551195s) [noise=true , address=77.90.31.17:4773 , piece_id=UB64CJMQSTLO4Z6P4VNFPFD3SCQ5HY6AQ5EDZXLCH3VNPZZTQRRA, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.943827442 (594h11m58.814910477s) [noise=true , address=145.79.227.45:4231 , piece_id=I7XS3UODLOAJORZ7K6BWX5GCJQBMFHWOYWDARUSQ34FRKSAFBKCA, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.946225057 (594h11m58.817308092s) [noise=true , address=45.91.116.13:4155 , piece_id=XDJXNXEY5EUHYOUAWI2EZ5WFKT7QYYSDX6P436HGA6AKAJYH5S5Q, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:40:58.948897196 (594h11m58.819980231s) [noise=true , address=145.79.227.9:4366 , piece_id=IGZ7H6AJYRANISEGHFRRN77ZZC6WWGHJM7RL4L3CP57GWKAL7AXQ, seg_pos={0 0}, upload=520892]: Uploading piece... Dec 30 09:41:00.630434513 (594h12m0.501517558s) [upload=520892]: Next returned nil segment Dec 30 09:41:00.630448059 (594h12m0.501531094s) [upload=520892]: Waiting for error group managing segments... Dec 30 09:41:00.910601240 (594h12m0.781684285s) [noise=true , address=145.79.227.6:4257 , piece_id=BTV2GMJ6UP74BJZLMYMIVRHXB7SQHHKOMJUZ2TCC3NI7YFYMOCQQ, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:00.920090881 (594h12m0.791173916s) [noise=true , address=145.79.227.32:4835 , piece_id=QV6KUS4JCIKX6RVNXVKOVVYGNL5DDL4JGXZYXOEV4F3KFQA5UZGQ, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:00.920803998 (594h12m0.791887033s) [noise=true , address=145.79.227.52:4446 , piece_id=CFN2DUXQZLD4KJCU3Q3YNDBLJEC4UQNKKRU7TNYYEYDUIXBPKMTA, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:00.921710415 (594h12m0.79279346s) [noise=true , address=145.79.227.36:4518 , piece_id=V6D77VUW7P5N7D6PJTVFB3MMP7QBOGUAXFIBLXVD76FGCU5JJI4Q, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:00.941792123 (594h12m0.812875168s) [noise=true , address=145.79.227.4:4185 , piece_id=QAN24H5MFCTSR7JF2VONIGQMKLPV365O547UHN5OV3DPUQCWRB4Q, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:00.945049606 (594h12m0.816132651s) [noise=true , address=145.79.227.25:4617 , piece_id=VRWMVGO656LO5KDP32ZPT65L7HR4YGA63HVL6BWWGIWT2HO2EM7A, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:00.956688779 (594h12m0.827771814s) [noise=true , address=145.79.227.23:4545 , piece_id=KCXQ6LTKZH454Q5NWWCPXPYX2WJWOHIDBYVEQWNPURXTFM5GYNFQ, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:01.023280592 (594h12m0.894363637s) [noise=true , address=145.79.227.45:4231 , piece_id=I7XS3UODLOAJORZ7K6BWX5GCJQBMFHWOYWDARUSQ34FRKSAFBKCA, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:01.033536321 (594h12m0.904619356s) [noise=true , address=145.79.227.9:4366 , piece_id=IGZ7H6AJYRANISEGHFRRN77ZZC6WWGHJM7RL4L3CP57GWKAL7AXQ, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:01.193569212 (594h12m1.064652257s) [noise=true , address=77.90.31.17:4773 , piece_id=UB64CJMQSTLO4Z6P4VNFPFD3SCQ5HY6AQ5EDZXLCH3VNPZZTQRRA, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:01.231586694 (594h12m1.102669739s) [noise=true , address=77.90.31.11:4554 , piece_id=XJNMRVSZFFZPYGWQMPC6WBX7VHEWD7HNIQB6UXW3SUP3KNY3ZJQQ, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:01.302711759 (594h12m1.173794804s) [noise=true , address=45.91.116.13:4155 , piece_id=XDJXNXEY5EUHYOUAWI2EZ5WFKT7QYYSDX6P436HGA6AKAJYH5S5Q, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:01.895766872 (594h12m1.766849907s) [noise=true , address=77.90.31.21:4878 , piece_id=RDB5KHAM4ZNUWC6HB6FDRSIZUQRSHYBXBWB66OBICTFBA5TKPMSQ, seg_pos={0 0}, upload=520892]: Done uploading piece. err: <nil> Dec 30 09:41:01.895795697 (594h12m1.766878742s) [seg_pos={0 0}, upload=520892]: Segment reached optimal threshold of 13 pieces. Dec 30 09:41:01.896047607 (594h12m1.767130662s) [noise=true , address=145.79.227.43:4157 , piece_id=DMTHBEXA4KFU3VUY24GA3IZ4CGMGAPAPHGVXPFMAT3KW5KCRX3GA, seg_pos={0 0}, upload=520892]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 09:41:01.896058228 (594h12m1.767141273s) [noise=true , address=145.79.227.43:4157 , piece_id=DMTHBEXA4KFU3VUY24GA3IZ4CGMGAPAPHGVXPFMAT3KW5KCRX3GA, seg_pos={0 0}, upload=520892]: Long tail cancellation due to optimal threshold Dec 30 09:41:01.896281753 (594h12m1.767364798s) [noise=true , address=45.91.116.8:4045 , piece_id=775ZABQ6Y6DFYWNWSNIETMTV6XUKC2ZCA6ZOQRBH34YIORFPJCIQ, seg_pos={0 0}, upload=520892]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 30 09:41:01.896300399 (594h12m1.767383444s) [noise=true , address=45.91.116.8:4045 , piece_id=775ZABQ6Y6DFYWNWSNIETMTV6XUKC2ZCA6ZOQRBH34YIORFPJCIQ, seg_pos={0 0}, upload=520892]: Long tail cancellation due to optimal threshold Dec 30 09:41:01.896464490 (594h12m1.767547536s) [noise=true , address=145.79.227.40:4051 , piece_id=7YYY4NGG3DG4LJS2HJU5JK5VFDJPI7MCRFNBCMY6FLMQAVEDN7IA, seg_pos={0 0}, upload=520892]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 09:41:01.896476674 (594h12m1.767559719s) [noise=true , address=145.79.227.40:4051 , piece_id=7YYY4NGG3DG4LJS2HJU5JK5VFDJPI7MCRFNBCMY6FLMQAVEDN7IA, seg_pos={0 0}, upload=520892]: Long tail cancellation due to optimal threshold Dec 30 09:41:01.896501381 (594h12m1.767584426s) [seg_pos={0 0}, upload=520892]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:41:01.896603906 (594h12m1.767686951s) [upload=520892]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:41:01.896612112 (594h12m1.767695157s) [upload=520892]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:41:02.186287346 (594h12m2.057370391s) [upload=520892]: Done uploading segments... Dec 30 09:41:02.186314738 (594h12m2.057397783s) [upload=520892]: Upload finished. err: <nil> Duration: 30.002791998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X....X...X..XX..XX......X...X...X...X...X..............X....] duration: 5.936027292s trace: https://jaeger.dev.storj.io/trace/1d49fa267a105226 FAIL (2025-12-30 09:40:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: JAGXQBFZAT47VYG6QPABG3VLF5GG5ZXXD7QZQVUUBZXDBOZIUTPQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: FQFAKYDGKF436U6LYDFULMTY5GGHRBBDJWNH2H5GBLOPRPYQRBCA) context deadline exceeded ======LOGS====== Dec 30 09:37:11.998184747 (594h8m11.869267782s) [upload=520837]: Starting upload Dec 30 09:37:11.998242858 (594h8m11.869325903s) [upload=520837]: Done starting upload Dec 30 09:37:11.998320736 (594h8m11.869403781s) [upload=520837]: Uploading segments... Dec 30 09:37:11.998430305 (594h8m11.86951334s) [upload=520837]: Got next segment. Inline: false Dec 30 09:37:11.998497683 (594h8m11.869580718s) [upload=520837]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:37:11.998503394 (594h8m11.869586429s) [upload=520837]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:37:12.245423483 (594h8m12.116506528s) [seg_pos={0 0}, upload=520837]: Begin upload segment... Dec 30 09:37:12.245489709 (594h8m12.116572754s) [seg_pos={0 0}, upload=520837]: Done begin upload segment. Dec 30 09:37:12.245517632 (594h8m12.116600678s) [noise=true , address=145.79.227.44:4195 , piece_id=4NY5NQA4EENHLLNVHSXYF4MND3PXNAZZMQ2BEU5AYPHBMOH52X5Q, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.245528613 (594h8m12.116611658s) [noise=true , address=45.91.116.12:4225 , piece_id=FQFAKYDGKF436U6LYDFULMTY5GGHRBBDJWNH2H5GBLOPRPYQRBCA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.245603135 (594h8m12.11668617s) [noise=true , address=145.79.227.42:4122 , piece_id=D54AIG6TEK6RYR5YRFRV6YYKK6GPNJKM4XXH27JGF2LHEND2EFRA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.246052551 (594h8m12.117135596s) [noise=true , address=145.79.227.47:4265 , piece_id=LYN4UELULFAVN35SSYBFNKNGPMPTPWUK5WTNIE7XAQXIDQNKB4MQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.246425802 (594h8m12.117508837s) [noise=true , address=45.91.116.13:4153 , piece_id=PSBTEW3FXGT7KCV3MOPTJUWOG4OHEAPHW3SL4LFP6R2UJSME2KKA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.246816255 (594h8m12.1178993s) [noise=true , address=145.79.227.25:4617 , piece_id=XHCPRRMPIFB6QUJ7VAGOK7FB57O3PKSXXTDIRQGOPHURYCC22PQA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.247202450 (594h8m12.118285495s) [noise=true , address=145.79.227.7:4294 , piece_id=DJBOJUYN7HQRVSJ2BIACUETF4R3PWZYBLMGR37KC42SGUMII7ENQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.247776564 (594h8m12.118859609s) [noise=true , address=145.79.227.27:4691 , piece_id=JIHOPPJTLAJ57NXWUAMFNXYCUHWRACD3W7AX52V3M5PUBVRNXQLA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.248267669 (594h8m12.119350714s) [noise=true , address=145.79.227.26:4653 , piece_id=THCXKJBYHIBGZVBYGADFTPSF65RAI7TVEXK3W45EW76LAZSTMOQA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.249255339 (594h8m12.120338384s) [noise=true , address=145.79.227.15:4876 , piece_id=LZZTIOEGI27YB2P6IPFFUEBTSMKPCI4QJA6QDAWMDXSBVOOOMHWA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.249920295 (594h8m12.12100334s) [noise=true , address=45.91.116.11:4117 , piece_id=VQHZGR5NJEMYTKLOJEXL6Q4AMJSIMU3MOULK4OBDAZTPQTJ4ODBQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.250646468 (594h8m12.121729503s) [noise=true , address=145.79.227.45:4231 , piece_id=GNS33UBQPHYRRX2Z5KVU3YHVQYBU6UIJ72FK3KVPVKOQE6FSW6QQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.251624090 (594h8m12.122707135s) [noise=true , address=45.91.116.10:4082 , piece_id=KBAB24TRI3LSPSH6FHG5T2LTSNCVRORMA7DFVKX7KKNNKAKK3INQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.251869908 (594h8m12.122952943s) [noise=true , address=45.91.116.5:4298 , piece_id=JAGXQBFZAT47VYG6QPABG3VLF5GG5ZXXD7QZQVUUBZXDBOZIUTPQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.253106672 (594h8m12.124189717s) [noise=true , address=145.79.227.16:4910 , piece_id=4DFEN2XQUPTBPKSEGQHVESAUKLA6BJBQWWHRM6RQ7BWBT6YJJW7Q, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.254630093 (594h8m12.125713128s) [noise=true , address=145.79.227.28:4726 , piece_id=DJWWNLC7KVY7BI6S2D2NHKAOH73SXJX6XO6K72T2DFLMJJ6SUOZQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:13.946908938 (594h8m13.817991983s) [upload=520837]: Next returned nil segment Dec 30 09:37:13.946921903 (594h8m13.818004938s) [upload=520837]: Waiting for error group managing segments... Dec 30 09:37:14.227917284 (594h8m14.099000329s) [noise=true , address=145.79.227.16:4910 , piece_id=4DFEN2XQUPTBPKSEGQHVESAUKLA6BJBQWWHRM6RQ7BWBT6YJJW7Q, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.239212040 (594h8m14.110295075s) [noise=true , address=145.79.227.15:4876 , piece_id=LZZTIOEGI27YB2P6IPFFUEBTSMKPCI4QJA6QDAWMDXSBVOOOMHWA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.241175229 (594h8m14.112258264s) [noise=true , address=145.79.227.28:4726 , piece_id=DJWWNLC7KVY7BI6S2D2NHKAOH73SXJX6XO6K72T2DFLMJJ6SUOZQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.243023449 (594h8m14.114106484s) [noise=true , address=145.79.227.42:4122 , piece_id=D54AIG6TEK6RYR5YRFRV6YYKK6GPNJKM4XXH27JGF2LHEND2EFRA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.246043018 (594h8m14.117126053s) [noise=true , address=145.79.227.25:4617 , piece_id=XHCPRRMPIFB6QUJ7VAGOK7FB57O3PKSXXTDIRQGOPHURYCC22PQA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.248630514 (594h8m14.119713549s) [noise=true , address=145.79.227.45:4231 , piece_id=GNS33UBQPHYRRX2Z5KVU3YHVQYBU6UIJ72FK3KVPVKOQE6FSW6QQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.262080175 (594h8m14.13316322s) [noise=true , address=145.79.227.7:4294 , piece_id=DJBOJUYN7HQRVSJ2BIACUETF4R3PWZYBLMGR37KC42SGUMII7ENQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.274266961 (594h8m14.145349996s) [noise=true , address=145.79.227.47:4265 , piece_id=LYN4UELULFAVN35SSYBFNKNGPMPTPWUK5WTNIE7XAQXIDQNKB4MQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.328127120 (594h8m14.199210165s) [noise=true , address=145.79.227.26:4653 , piece_id=THCXKJBYHIBGZVBYGADFTPSF65RAI7TVEXK3W45EW76LAZSTMOQA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.331722776 (594h8m14.202805811s) [noise=true , address=145.79.227.27:4691 , piece_id=JIHOPPJTLAJ57NXWUAMFNXYCUHWRACD3W7AX52V3M5PUBVRNXQLA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.580768008 (594h8m14.451851053s) [noise=true , address=45.91.116.5:4298 , piece_id=JAGXQBFZAT47VYG6QPABG3VLF5GG5ZXXD7QZQVUUBZXDBOZIUTPQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.663882332 (594h8m14.534965367s) [noise=true , address=45.91.116.12:4225 , piece_id=FQFAKYDGKF436U6LYDFULMTY5GGHRBBDJWNH2H5GBLOPRPYQRBCA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.691478029 (594h8m14.562561064s) [noise=true , address=45.91.116.11:4117 , piece_id=VQHZGR5NJEMYTKLOJEXL6Q4AMJSIMU3MOULK4OBDAZTPQTJ4ODBQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.691496694 (594h8m14.562579729s) [seg_pos={0 0}, upload=520837]: Segment reached optimal threshold of 13 pieces. Dec 30 09:37:14.691839757 (594h8m14.562922802s) [noise=true , address=45.91.116.13:4153 , piece_id=PSBTEW3FXGT7KCV3MOPTJUWOG4OHEAPHW3SL4LFP6R2UJSME2KKA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 09:37:14.691849455 (594h8m14.56293249s) [noise=true , address=45.91.116.13:4153 , piece_id=PSBTEW3FXGT7KCV3MOPTJUWOG4OHEAPHW3SL4LFP6R2UJSME2KKA, seg_pos={0 0}, upload=520837]: Long tail cancellation due to optimal threshold Dec 30 09:37:14.691862962 (594h8m14.562945997s) [noise=true , address=45.91.116.10:4082 , piece_id=KBAB24TRI3LSPSH6FHG5T2LTSNCVRORMA7DFVKX7KKNNKAKK3INQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:60096->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:60096->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60096->45.91.116.10:4082: use of closed network connection Dec 30 09:37:14.691874063 (594h8m14.562957098s) [noise=true , address=45.91.116.10:4082 , piece_id=KBAB24TRI3LSPSH6FHG5T2LTSNCVRORMA7DFVKX7KKNNKAKK3INQ, seg_pos={0 0}, upload=520837]: Long tail cancellation due to optimal threshold Dec 30 09:37:14.691990314 (594h8m14.563073349s) [noise=true , address=145.79.227.44:4195 , piece_id=4NY5NQA4EENHLLNVHSXYF4MND3PXNAZZMQ2BEU5AYPHBMOH52X5Q, seg_pos={0 0}, upload=520837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 09:37:14.691999542 (594h8m14.563082587s) [noise=true , address=145.79.227.44:4195 , piece_id=4NY5NQA4EENHLLNVHSXYF4MND3PXNAZZMQ2BEU5AYPHBMOH52X5Q, seg_pos={0 0}, upload=520837]: Long tail cancellation due to optimal threshold Dec 30 09:37:14.692022204 (594h8m14.563105239s) [seg_pos={0 0}, upload=520837]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:37:14.692128346 (594h8m14.563211392s) [upload=520837]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:37:14.692135341 (594h8m14.563218386s) [upload=520837]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:37:14.990189192 (594h8m14.861272227s) [upload=520837]: Done uploading segments... Dec 30 09:37:14.990207287 (594h8m14.861290322s) [upload=520837]: Upload finished. err: <nil> Duration: 5.93604763s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X....X...X..XX..XX......X...X...X...X...X..............X...] duration: 27.118106572s trace: https://jaeger.dev.storj.io/trace/1d49fa267a105226 FAIL (2025-12-30 09:40:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: JAGXQBFZAT47VYG6QPABG3VLF5GG5ZXXD7QZQVUUBZXDBOZIUTPQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: FQFAKYDGKF436U6LYDFULMTY5GGHRBBDJWNH2H5GBLOPRPYQRBCA) context deadline exceeded ======LOGS====== Dec 30 09:37:11.998184747 (594h8m11.869267782s) [upload=520837]: Starting upload Dec 30 09:37:11.998242858 (594h8m11.869325903s) [upload=520837]: Done starting upload Dec 30 09:37:11.998320736 (594h8m11.869403781s) [upload=520837]: Uploading segments... Dec 30 09:37:11.998430305 (594h8m11.86951334s) [upload=520837]: Got next segment. Inline: false Dec 30 09:37:11.998497683 (594h8m11.869580718s) [upload=520837]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:37:11.998503394 (594h8m11.869586429s) [upload=520837]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:37:12.245423483 (594h8m12.116506528s) [seg_pos={0 0}, upload=520837]: Begin upload segment... Dec 30 09:37:12.245489709 (594h8m12.116572754s) [seg_pos={0 0}, upload=520837]: Done begin upload segment. Dec 30 09:37:12.245517632 (594h8m12.116600678s) [noise=true , address=145.79.227.44:4195 , piece_id=4NY5NQA4EENHLLNVHSXYF4MND3PXNAZZMQ2BEU5AYPHBMOH52X5Q, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.245528613 (594h8m12.116611658s) [noise=true , address=45.91.116.12:4225 , piece_id=FQFAKYDGKF436U6LYDFULMTY5GGHRBBDJWNH2H5GBLOPRPYQRBCA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.245603135 (594h8m12.11668617s) [noise=true , address=145.79.227.42:4122 , piece_id=D54AIG6TEK6RYR5YRFRV6YYKK6GPNJKM4XXH27JGF2LHEND2EFRA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.246052551 (594h8m12.117135596s) [noise=true , address=145.79.227.47:4265 , piece_id=LYN4UELULFAVN35SSYBFNKNGPMPTPWUK5WTNIE7XAQXIDQNKB4MQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.246425802 (594h8m12.117508837s) [noise=true , address=45.91.116.13:4153 , piece_id=PSBTEW3FXGT7KCV3MOPTJUWOG4OHEAPHW3SL4LFP6R2UJSME2KKA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.246816255 (594h8m12.1178993s) [noise=true , address=145.79.227.25:4617 , piece_id=XHCPRRMPIFB6QUJ7VAGOK7FB57O3PKSXXTDIRQGOPHURYCC22PQA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.247202450 (594h8m12.118285495s) [noise=true , address=145.79.227.7:4294 , piece_id=DJBOJUYN7HQRVSJ2BIACUETF4R3PWZYBLMGR37KC42SGUMII7ENQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.247776564 (594h8m12.118859609s) [noise=true , address=145.79.227.27:4691 , piece_id=JIHOPPJTLAJ57NXWUAMFNXYCUHWRACD3W7AX52V3M5PUBVRNXQLA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.248267669 (594h8m12.119350714s) [noise=true , address=145.79.227.26:4653 , piece_id=THCXKJBYHIBGZVBYGADFTPSF65RAI7TVEXK3W45EW76LAZSTMOQA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.249255339 (594h8m12.120338384s) [noise=true , address=145.79.227.15:4876 , piece_id=LZZTIOEGI27YB2P6IPFFUEBTSMKPCI4QJA6QDAWMDXSBVOOOMHWA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.249920295 (594h8m12.12100334s) [noise=true , address=45.91.116.11:4117 , piece_id=VQHZGR5NJEMYTKLOJEXL6Q4AMJSIMU3MOULK4OBDAZTPQTJ4ODBQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.250646468 (594h8m12.121729503s) [noise=true , address=145.79.227.45:4231 , piece_id=GNS33UBQPHYRRX2Z5KVU3YHVQYBU6UIJ72FK3KVPVKOQE6FSW6QQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.251624090 (594h8m12.122707135s) [noise=true , address=45.91.116.10:4082 , piece_id=KBAB24TRI3LSPSH6FHG5T2LTSNCVRORMA7DFVKX7KKNNKAKK3INQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.251869908 (594h8m12.122952943s) [noise=true , address=45.91.116.5:4298 , piece_id=JAGXQBFZAT47VYG6QPABG3VLF5GG5ZXXD7QZQVUUBZXDBOZIUTPQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.253106672 (594h8m12.124189717s) [noise=true , address=145.79.227.16:4910 , piece_id=4DFEN2XQUPTBPKSEGQHVESAUKLA6BJBQWWHRM6RQ7BWBT6YJJW7Q, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.254630093 (594h8m12.125713128s) [noise=true , address=145.79.227.28:4726 , piece_id=DJWWNLC7KVY7BI6S2D2NHKAOH73SXJX6XO6K72T2DFLMJJ6SUOZQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:13.946908938 (594h8m13.817991983s) [upload=520837]: Next returned nil segment Dec 30 09:37:13.946921903 (594h8m13.818004938s) [upload=520837]: Waiting for error group managing segments... Dec 30 09:37:14.227917284 (594h8m14.099000329s) [noise=true , address=145.79.227.16:4910 , piece_id=4DFEN2XQUPTBPKSEGQHVESAUKLA6BJBQWWHRM6RQ7BWBT6YJJW7Q, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.239212040 (594h8m14.110295075s) [noise=true , address=145.79.227.15:4876 , piece_id=LZZTIOEGI27YB2P6IPFFUEBTSMKPCI4QJA6QDAWMDXSBVOOOMHWA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.241175229 (594h8m14.112258264s) [noise=true , address=145.79.227.28:4726 , piece_id=DJWWNLC7KVY7BI6S2D2NHKAOH73SXJX6XO6K72T2DFLMJJ6SUOZQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.243023449 (594h8m14.114106484s) [noise=true , address=145.79.227.42:4122 , piece_id=D54AIG6TEK6RYR5YRFRV6YYKK6GPNJKM4XXH27JGF2LHEND2EFRA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.246043018 (594h8m14.117126053s) [noise=true , address=145.79.227.25:4617 , piece_id=XHCPRRMPIFB6QUJ7VAGOK7FB57O3PKSXXTDIRQGOPHURYCC22PQA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.248630514 (594h8m14.119713549s) [noise=true , address=145.79.227.45:4231 , piece_id=GNS33UBQPHYRRX2Z5KVU3YHVQYBU6UIJ72FK3KVPVKOQE6FSW6QQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.262080175 (594h8m14.13316322s) [noise=true , address=145.79.227.7:4294 , piece_id=DJBOJUYN7HQRVSJ2BIACUETF4R3PWZYBLMGR37KC42SGUMII7ENQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.274266961 (594h8m14.145349996s) [noise=true , address=145.79.227.47:4265 , piece_id=LYN4UELULFAVN35SSYBFNKNGPMPTPWUK5WTNIE7XAQXIDQNKB4MQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.328127120 (594h8m14.199210165s) [noise=true , address=145.79.227.26:4653 , piece_id=THCXKJBYHIBGZVBYGADFTPSF65RAI7TVEXK3W45EW76LAZSTMOQA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.331722776 (594h8m14.202805811s) [noise=true , address=145.79.227.27:4691 , piece_id=JIHOPPJTLAJ57NXWUAMFNXYCUHWRACD3W7AX52V3M5PUBVRNXQLA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.580768008 (594h8m14.451851053s) [noise=true , address=45.91.116.5:4298 , piece_id=JAGXQBFZAT47VYG6QPABG3VLF5GG5ZXXD7QZQVUUBZXDBOZIUTPQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.663882332 (594h8m14.534965367s) [noise=true , address=45.91.116.12:4225 , piece_id=FQFAKYDGKF436U6LYDFULMTY5GGHRBBDJWNH2H5GBLOPRPYQRBCA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.691478029 (594h8m14.562561064s) [noise=true , address=45.91.116.11:4117 , piece_id=VQHZGR5NJEMYTKLOJEXL6Q4AMJSIMU3MOULK4OBDAZTPQTJ4ODBQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.691496694 (594h8m14.562579729s) [seg_pos={0 0}, upload=520837]: Segment reached optimal threshold of 13 pieces. Dec 30 09:37:14.691839757 (594h8m14.562922802s) [noise=true , address=45.91.116.13:4153 , piece_id=PSBTEW3FXGT7KCV3MOPTJUWOG4OHEAPHW3SL4LFP6R2UJSME2KKA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 09:37:14.691849455 (594h8m14.56293249s) [noise=true , address=45.91.116.13:4153 , piece_id=PSBTEW3FXGT7KCV3MOPTJUWOG4OHEAPHW3SL4LFP6R2UJSME2KKA, seg_pos={0 0}, upload=520837]: Long tail cancellation due to optimal threshold Dec 30 09:37:14.691862962 (594h8m14.562945997s) [noise=true , address=45.91.116.10:4082 , piece_id=KBAB24TRI3LSPSH6FHG5T2LTSNCVRORMA7DFVKX7KKNNKAKK3INQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:60096->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:60096->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60096->45.91.116.10:4082: use of closed network connection Dec 30 09:37:14.691874063 (594h8m14.562957098s) [noise=true , address=45.91.116.10:4082 , piece_id=KBAB24TRI3LSPSH6FHG5T2LTSNCVRORMA7DFVKX7KKNNKAKK3INQ, seg_pos={0 0}, upload=520837]: Long tail cancellation due to optimal threshold Dec 30 09:37:14.691990314 (594h8m14.563073349s) [noise=true , address=145.79.227.44:4195 , piece_id=4NY5NQA4EENHLLNVHSXYF4MND3PXNAZZMQ2BEU5AYPHBMOH52X5Q, seg_pos={0 0}, upload=520837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 09:37:14.691999542 (594h8m14.563082587s) [noise=true , address=145.79.227.44:4195 , piece_id=4NY5NQA4EENHLLNVHSXYF4MND3PXNAZZMQ2BEU5AYPHBMOH52X5Q, seg_pos={0 0}, upload=520837]: Long tail cancellation due to optimal threshold Dec 30 09:37:14.692022204 (594h8m14.563105239s) [seg_pos={0 0}, upload=520837]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:37:14.692128346 (594h8m14.563211392s) [upload=520837]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:37:14.692135341 (594h8m14.563218386s) [upload=520837]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:37:14.990189192 (594h8m14.861272227s) [upload=520837]: Done uploading segments... Dec 30 09:37:14.990207287 (594h8m14.861290322s) [upload=520837]: Upload finished. err: <nil> Duration: 27.118140226s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.X.X....X.X...X........X.....X....XXX..X.X.X.......XX....X..] duration: 4.173160265s trace: https://jaeger.dev.storj.io/trace/1876eff5543ce594 FAIL (2025-12-30 09:40:26 UTC) orbiter warning: context deadline exceeded Duration: 4.173176617s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..........................X...............................] duration: 39.627885363s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:40:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 39.627916161s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [..X.X....X.X...X........X.....X....XXX..X.X.X.......XX....X.] duration: 3.577755341s trace: https://jaeger.dev.storj.io/trace/1876eff5543ce594 FAIL (2025-12-30 09:39:39 UTC) orbiter warning: context deadline exceeded Duration: 3.577848477s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X....X...X..XX..XX......X...X...X...X...X..............X..] duration: 6.052467185s trace: https://jaeger.dev.storj.io/trace/1d49fa267a105226 FAIL (2025-12-30 09:39:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: JAGXQBFZAT47VYG6QPABG3VLF5GG5ZXXD7QZQVUUBZXDBOZIUTPQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: FQFAKYDGKF436U6LYDFULMTY5GGHRBBDJWNH2H5GBLOPRPYQRBCA) context deadline exceeded ======LOGS====== Dec 30 09:37:11.998184747 (594h8m11.869267782s) [upload=520837]: Starting upload Dec 30 09:37:11.998242858 (594h8m11.869325903s) [upload=520837]: Done starting upload Dec 30 09:37:11.998320736 (594h8m11.869403781s) [upload=520837]: Uploading segments... Dec 30 09:37:11.998430305 (594h8m11.86951334s) [upload=520837]: Got next segment. Inline: false Dec 30 09:37:11.998497683 (594h8m11.869580718s) [upload=520837]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:37:11.998503394 (594h8m11.869586429s) [upload=520837]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:37:12.245423483 (594h8m12.116506528s) [seg_pos={0 0}, upload=520837]: Begin upload segment... Dec 30 09:37:12.245489709 (594h8m12.116572754s) [seg_pos={0 0}, upload=520837]: Done begin upload segment. Dec 30 09:37:12.245517632 (594h8m12.116600678s) [noise=true , address=145.79.227.44:4195 , piece_id=4NY5NQA4EENHLLNVHSXYF4MND3PXNAZZMQ2BEU5AYPHBMOH52X5Q, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.245528613 (594h8m12.116611658s) [noise=true , address=45.91.116.12:4225 , piece_id=FQFAKYDGKF436U6LYDFULMTY5GGHRBBDJWNH2H5GBLOPRPYQRBCA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.245603135 (594h8m12.11668617s) [noise=true , address=145.79.227.42:4122 , piece_id=D54AIG6TEK6RYR5YRFRV6YYKK6GPNJKM4XXH27JGF2LHEND2EFRA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.246052551 (594h8m12.117135596s) [noise=true , address=145.79.227.47:4265 , piece_id=LYN4UELULFAVN35SSYBFNKNGPMPTPWUK5WTNIE7XAQXIDQNKB4MQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.246425802 (594h8m12.117508837s) [noise=true , address=45.91.116.13:4153 , piece_id=PSBTEW3FXGT7KCV3MOPTJUWOG4OHEAPHW3SL4LFP6R2UJSME2KKA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.246816255 (594h8m12.1178993s) [noise=true , address=145.79.227.25:4617 , piece_id=XHCPRRMPIFB6QUJ7VAGOK7FB57O3PKSXXTDIRQGOPHURYCC22PQA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.247202450 (594h8m12.118285495s) [noise=true , address=145.79.227.7:4294 , piece_id=DJBOJUYN7HQRVSJ2BIACUETF4R3PWZYBLMGR37KC42SGUMII7ENQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.247776564 (594h8m12.118859609s) [noise=true , address=145.79.227.27:4691 , piece_id=JIHOPPJTLAJ57NXWUAMFNXYCUHWRACD3W7AX52V3M5PUBVRNXQLA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.248267669 (594h8m12.119350714s) [noise=true , address=145.79.227.26:4653 , piece_id=THCXKJBYHIBGZVBYGADFTPSF65RAI7TVEXK3W45EW76LAZSTMOQA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.249255339 (594h8m12.120338384s) [noise=true , address=145.79.227.15:4876 , piece_id=LZZTIOEGI27YB2P6IPFFUEBTSMKPCI4QJA6QDAWMDXSBVOOOMHWA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.249920295 (594h8m12.12100334s) [noise=true , address=45.91.116.11:4117 , piece_id=VQHZGR5NJEMYTKLOJEXL6Q4AMJSIMU3MOULK4OBDAZTPQTJ4ODBQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.250646468 (594h8m12.121729503s) [noise=true , address=145.79.227.45:4231 , piece_id=GNS33UBQPHYRRX2Z5KVU3YHVQYBU6UIJ72FK3KVPVKOQE6FSW6QQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.251624090 (594h8m12.122707135s) [noise=true , address=45.91.116.10:4082 , piece_id=KBAB24TRI3LSPSH6FHG5T2LTSNCVRORMA7DFVKX7KKNNKAKK3INQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.251869908 (594h8m12.122952943s) [noise=true , address=45.91.116.5:4298 , piece_id=JAGXQBFZAT47VYG6QPABG3VLF5GG5ZXXD7QZQVUUBZXDBOZIUTPQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.253106672 (594h8m12.124189717s) [noise=true , address=145.79.227.16:4910 , piece_id=4DFEN2XQUPTBPKSEGQHVESAUKLA6BJBQWWHRM6RQ7BWBT6YJJW7Q, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.254630093 (594h8m12.125713128s) [noise=true , address=145.79.227.28:4726 , piece_id=DJWWNLC7KVY7BI6S2D2NHKAOH73SXJX6XO6K72T2DFLMJJ6SUOZQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:13.946908938 (594h8m13.817991983s) [upload=520837]: Next returned nil segment Dec 30 09:37:13.946921903 (594h8m13.818004938s) [upload=520837]: Waiting for error group managing segments... Dec 30 09:37:14.227917284 (594h8m14.099000329s) [noise=true , address=145.79.227.16:4910 , piece_id=4DFEN2XQUPTBPKSEGQHVESAUKLA6BJBQWWHRM6RQ7BWBT6YJJW7Q, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.239212040 (594h8m14.110295075s) [noise=true , address=145.79.227.15:4876 , piece_id=LZZTIOEGI27YB2P6IPFFUEBTSMKPCI4QJA6QDAWMDXSBVOOOMHWA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.241175229 (594h8m14.112258264s) [noise=true , address=145.79.227.28:4726 , piece_id=DJWWNLC7KVY7BI6S2D2NHKAOH73SXJX6XO6K72T2DFLMJJ6SUOZQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.243023449 (594h8m14.114106484s) [noise=true , address=145.79.227.42:4122 , piece_id=D54AIG6TEK6RYR5YRFRV6YYKK6GPNJKM4XXH27JGF2LHEND2EFRA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.246043018 (594h8m14.117126053s) [noise=true , address=145.79.227.25:4617 , piece_id=XHCPRRMPIFB6QUJ7VAGOK7FB57O3PKSXXTDIRQGOPHURYCC22PQA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.248630514 (594h8m14.119713549s) [noise=true , address=145.79.227.45:4231 , piece_id=GNS33UBQPHYRRX2Z5KVU3YHVQYBU6UIJ72FK3KVPVKOQE6FSW6QQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.262080175 (594h8m14.13316322s) [noise=true , address=145.79.227.7:4294 , piece_id=DJBOJUYN7HQRVSJ2BIACUETF4R3PWZYBLMGR37KC42SGUMII7ENQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.274266961 (594h8m14.145349996s) [noise=true , address=145.79.227.47:4265 , piece_id=LYN4UELULFAVN35SSYBFNKNGPMPTPWUK5WTNIE7XAQXIDQNKB4MQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.328127120 (594h8m14.199210165s) [noise=true , address=145.79.227.26:4653 , piece_id=THCXKJBYHIBGZVBYGADFTPSF65RAI7TVEXK3W45EW76LAZSTMOQA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.331722776 (594h8m14.202805811s) [noise=true , address=145.79.227.27:4691 , piece_id=JIHOPPJTLAJ57NXWUAMFNXYCUHWRACD3W7AX52V3M5PUBVRNXQLA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.580768008 (594h8m14.451851053s) [noise=true , address=45.91.116.5:4298 , piece_id=JAGXQBFZAT47VYG6QPABG3VLF5GG5ZXXD7QZQVUUBZXDBOZIUTPQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.663882332 (594h8m14.534965367s) [noise=true , address=45.91.116.12:4225 , piece_id=FQFAKYDGKF436U6LYDFULMTY5GGHRBBDJWNH2H5GBLOPRPYQRBCA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.691478029 (594h8m14.562561064s) [noise=true , address=45.91.116.11:4117 , piece_id=VQHZGR5NJEMYTKLOJEXL6Q4AMJSIMU3MOULK4OBDAZTPQTJ4ODBQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.691496694 (594h8m14.562579729s) [seg_pos={0 0}, upload=520837]: Segment reached optimal threshold of 13 pieces. Dec 30 09:37:14.691839757 (594h8m14.562922802s) [noise=true , address=45.91.116.13:4153 , piece_id=PSBTEW3FXGT7KCV3MOPTJUWOG4OHEAPHW3SL4LFP6R2UJSME2KKA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 09:37:14.691849455 (594h8m14.56293249s) [noise=true , address=45.91.116.13:4153 , piece_id=PSBTEW3FXGT7KCV3MOPTJUWOG4OHEAPHW3SL4LFP6R2UJSME2KKA, seg_pos={0 0}, upload=520837]: Long tail cancellation due to optimal threshold Dec 30 09:37:14.691862962 (594h8m14.562945997s) [noise=true , address=45.91.116.10:4082 , piece_id=KBAB24TRI3LSPSH6FHG5T2LTSNCVRORMA7DFVKX7KKNNKAKK3INQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:60096->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:60096->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60096->45.91.116.10:4082: use of closed network connection Dec 30 09:37:14.691874063 (594h8m14.562957098s) [noise=true , address=45.91.116.10:4082 , piece_id=KBAB24TRI3LSPSH6FHG5T2LTSNCVRORMA7DFVKX7KKNNKAKK3INQ, seg_pos={0 0}, upload=520837]: Long tail cancellation due to optimal threshold Dec 30 09:37:14.691990314 (594h8m14.563073349s) [noise=true , address=145.79.227.44:4195 , piece_id=4NY5NQA4EENHLLNVHSXYF4MND3PXNAZZMQ2BEU5AYPHBMOH52X5Q, seg_pos={0 0}, upload=520837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 09:37:14.691999542 (594h8m14.563082587s) [noise=true , address=145.79.227.44:4195 , piece_id=4NY5NQA4EENHLLNVHSXYF4MND3PXNAZZMQ2BEU5AYPHBMOH52X5Q, seg_pos={0 0}, upload=520837]: Long tail cancellation due to optimal threshold Dec 30 09:37:14.692022204 (594h8m14.563105239s) [seg_pos={0 0}, upload=520837]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:37:14.692128346 (594h8m14.563211392s) [upload=520837]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:37:14.692135341 (594h8m14.563218386s) [upload=520837]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:37:14.990189192 (594h8m14.861272227s) [upload=520837]: Done uploading segments... Dec 30 09:37:14.990207287 (594h8m14.861290322s) [upload=520837]: Upload finished. err: <nil> Duration: 6.052502643s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..........................X..............................] duration: 32.541594713s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:39:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 32.541631643s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [...X.X....X.X...X........X.....X....XXX..X.X.X.......XX....X] duration: 5.001408798s trace: https://jaeger.dev.storj.io/trace/1876eff5543ce594 FAIL (2025-12-30 09:38:51 UTC) orbiter warning: context deadline exceeded Duration: 5.00143035s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X....X...X..XX..XX......X...X...X...X...X..............X.] duration: 6.105687301s trace: https://jaeger.dev.storj.io/trace/1d49fa267a105226 FAIL (2025-12-30 09:38:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: JAGXQBFZAT47VYG6QPABG3VLF5GG5ZXXD7QZQVUUBZXDBOZIUTPQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: FQFAKYDGKF436U6LYDFULMTY5GGHRBBDJWNH2H5GBLOPRPYQRBCA) context deadline exceeded ======LOGS====== Dec 30 09:37:11.998184747 (594h8m11.869267782s) [upload=520837]: Starting upload Dec 30 09:37:11.998242858 (594h8m11.869325903s) [upload=520837]: Done starting upload Dec 30 09:37:11.998320736 (594h8m11.869403781s) [upload=520837]: Uploading segments... Dec 30 09:37:11.998430305 (594h8m11.86951334s) [upload=520837]: Got next segment. Inline: false Dec 30 09:37:11.998497683 (594h8m11.869580718s) [upload=520837]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:37:11.998503394 (594h8m11.869586429s) [upload=520837]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:37:12.245423483 (594h8m12.116506528s) [seg_pos={0 0}, upload=520837]: Begin upload segment... Dec 30 09:37:12.245489709 (594h8m12.116572754s) [seg_pos={0 0}, upload=520837]: Done begin upload segment. Dec 30 09:37:12.245517632 (594h8m12.116600678s) [noise=true , address=145.79.227.44:4195 , piece_id=4NY5NQA4EENHLLNVHSXYF4MND3PXNAZZMQ2BEU5AYPHBMOH52X5Q, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.245528613 (594h8m12.116611658s) [noise=true , address=45.91.116.12:4225 , piece_id=FQFAKYDGKF436U6LYDFULMTY5GGHRBBDJWNH2H5GBLOPRPYQRBCA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.245603135 (594h8m12.11668617s) [noise=true , address=145.79.227.42:4122 , piece_id=D54AIG6TEK6RYR5YRFRV6YYKK6GPNJKM4XXH27JGF2LHEND2EFRA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.246052551 (594h8m12.117135596s) [noise=true , address=145.79.227.47:4265 , piece_id=LYN4UELULFAVN35SSYBFNKNGPMPTPWUK5WTNIE7XAQXIDQNKB4MQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.246425802 (594h8m12.117508837s) [noise=true , address=45.91.116.13:4153 , piece_id=PSBTEW3FXGT7KCV3MOPTJUWOG4OHEAPHW3SL4LFP6R2UJSME2KKA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.246816255 (594h8m12.1178993s) [noise=true , address=145.79.227.25:4617 , piece_id=XHCPRRMPIFB6QUJ7VAGOK7FB57O3PKSXXTDIRQGOPHURYCC22PQA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.247202450 (594h8m12.118285495s) [noise=true , address=145.79.227.7:4294 , piece_id=DJBOJUYN7HQRVSJ2BIACUETF4R3PWZYBLMGR37KC42SGUMII7ENQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.247776564 (594h8m12.118859609s) [noise=true , address=145.79.227.27:4691 , piece_id=JIHOPPJTLAJ57NXWUAMFNXYCUHWRACD3W7AX52V3M5PUBVRNXQLA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.248267669 (594h8m12.119350714s) [noise=true , address=145.79.227.26:4653 , piece_id=THCXKJBYHIBGZVBYGADFTPSF65RAI7TVEXK3W45EW76LAZSTMOQA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.249255339 (594h8m12.120338384s) [noise=true , address=145.79.227.15:4876 , piece_id=LZZTIOEGI27YB2P6IPFFUEBTSMKPCI4QJA6QDAWMDXSBVOOOMHWA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.249920295 (594h8m12.12100334s) [noise=true , address=45.91.116.11:4117 , piece_id=VQHZGR5NJEMYTKLOJEXL6Q4AMJSIMU3MOULK4OBDAZTPQTJ4ODBQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.250646468 (594h8m12.121729503s) [noise=true , address=145.79.227.45:4231 , piece_id=GNS33UBQPHYRRX2Z5KVU3YHVQYBU6UIJ72FK3KVPVKOQE6FSW6QQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.251624090 (594h8m12.122707135s) [noise=true , address=45.91.116.10:4082 , piece_id=KBAB24TRI3LSPSH6FHG5T2LTSNCVRORMA7DFVKX7KKNNKAKK3INQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.251869908 (594h8m12.122952943s) [noise=true , address=45.91.116.5:4298 , piece_id=JAGXQBFZAT47VYG6QPABG3VLF5GG5ZXXD7QZQVUUBZXDBOZIUTPQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.253106672 (594h8m12.124189717s) [noise=true , address=145.79.227.16:4910 , piece_id=4DFEN2XQUPTBPKSEGQHVESAUKLA6BJBQWWHRM6RQ7BWBT6YJJW7Q, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.254630093 (594h8m12.125713128s) [noise=true , address=145.79.227.28:4726 , piece_id=DJWWNLC7KVY7BI6S2D2NHKAOH73SXJX6XO6K72T2DFLMJJ6SUOZQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:13.946908938 (594h8m13.817991983s) [upload=520837]: Next returned nil segment Dec 30 09:37:13.946921903 (594h8m13.818004938s) [upload=520837]: Waiting for error group managing segments... Dec 30 09:37:14.227917284 (594h8m14.099000329s) [noise=true , address=145.79.227.16:4910 , piece_id=4DFEN2XQUPTBPKSEGQHVESAUKLA6BJBQWWHRM6RQ7BWBT6YJJW7Q, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.239212040 (594h8m14.110295075s) [noise=true , address=145.79.227.15:4876 , piece_id=LZZTIOEGI27YB2P6IPFFUEBTSMKPCI4QJA6QDAWMDXSBVOOOMHWA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.241175229 (594h8m14.112258264s) [noise=true , address=145.79.227.28:4726 , piece_id=DJWWNLC7KVY7BI6S2D2NHKAOH73SXJX6XO6K72T2DFLMJJ6SUOZQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.243023449 (594h8m14.114106484s) [noise=true , address=145.79.227.42:4122 , piece_id=D54AIG6TEK6RYR5YRFRV6YYKK6GPNJKM4XXH27JGF2LHEND2EFRA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.246043018 (594h8m14.117126053s) [noise=true , address=145.79.227.25:4617 , piece_id=XHCPRRMPIFB6QUJ7VAGOK7FB57O3PKSXXTDIRQGOPHURYCC22PQA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.248630514 (594h8m14.119713549s) [noise=true , address=145.79.227.45:4231 , piece_id=GNS33UBQPHYRRX2Z5KVU3YHVQYBU6UIJ72FK3KVPVKOQE6FSW6QQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.262080175 (594h8m14.13316322s) [noise=true , address=145.79.227.7:4294 , piece_id=DJBOJUYN7HQRVSJ2BIACUETF4R3PWZYBLMGR37KC42SGUMII7ENQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.274266961 (594h8m14.145349996s) [noise=true , address=145.79.227.47:4265 , piece_id=LYN4UELULFAVN35SSYBFNKNGPMPTPWUK5WTNIE7XAQXIDQNKB4MQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.328127120 (594h8m14.199210165s) [noise=true , address=145.79.227.26:4653 , piece_id=THCXKJBYHIBGZVBYGADFTPSF65RAI7TVEXK3W45EW76LAZSTMOQA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.331722776 (594h8m14.202805811s) [noise=true , address=145.79.227.27:4691 , piece_id=JIHOPPJTLAJ57NXWUAMFNXYCUHWRACD3W7AX52V3M5PUBVRNXQLA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.580768008 (594h8m14.451851053s) [noise=true , address=45.91.116.5:4298 , piece_id=JAGXQBFZAT47VYG6QPABG3VLF5GG5ZXXD7QZQVUUBZXDBOZIUTPQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.663882332 (594h8m14.534965367s) [noise=true , address=45.91.116.12:4225 , piece_id=FQFAKYDGKF436U6LYDFULMTY5GGHRBBDJWNH2H5GBLOPRPYQRBCA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.691478029 (594h8m14.562561064s) [noise=true , address=45.91.116.11:4117 , piece_id=VQHZGR5NJEMYTKLOJEXL6Q4AMJSIMU3MOULK4OBDAZTPQTJ4ODBQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.691496694 (594h8m14.562579729s) [seg_pos={0 0}, upload=520837]: Segment reached optimal threshold of 13 pieces. Dec 30 09:37:14.691839757 (594h8m14.562922802s) [noise=true , address=45.91.116.13:4153 , piece_id=PSBTEW3FXGT7KCV3MOPTJUWOG4OHEAPHW3SL4LFP6R2UJSME2KKA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 09:37:14.691849455 (594h8m14.56293249s) [noise=true , address=45.91.116.13:4153 , piece_id=PSBTEW3FXGT7KCV3MOPTJUWOG4OHEAPHW3SL4LFP6R2UJSME2KKA, seg_pos={0 0}, upload=520837]: Long tail cancellation due to optimal threshold Dec 30 09:37:14.691862962 (594h8m14.562945997s) [noise=true , address=45.91.116.10:4082 , piece_id=KBAB24TRI3LSPSH6FHG5T2LTSNCVRORMA7DFVKX7KKNNKAKK3INQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:60096->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:60096->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60096->45.91.116.10:4082: use of closed network connection Dec 30 09:37:14.691874063 (594h8m14.562957098s) [noise=true , address=45.91.116.10:4082 , piece_id=KBAB24TRI3LSPSH6FHG5T2LTSNCVRORMA7DFVKX7KKNNKAKK3INQ, seg_pos={0 0}, upload=520837]: Long tail cancellation due to optimal threshold Dec 30 09:37:14.691990314 (594h8m14.563073349s) [noise=true , address=145.79.227.44:4195 , piece_id=4NY5NQA4EENHLLNVHSXYF4MND3PXNAZZMQ2BEU5AYPHBMOH52X5Q, seg_pos={0 0}, upload=520837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 09:37:14.691999542 (594h8m14.563082587s) [noise=true , address=145.79.227.44:4195 , piece_id=4NY5NQA4EENHLLNVHSXYF4MND3PXNAZZMQ2BEU5AYPHBMOH52X5Q, seg_pos={0 0}, upload=520837]: Long tail cancellation due to optimal threshold Dec 30 09:37:14.692022204 (594h8m14.563105239s) [seg_pos={0 0}, upload=520837]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:37:14.692128346 (594h8m14.563211392s) [upload=520837]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:37:14.692135341 (594h8m14.563218386s) [upload=520837]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:37:14.990189192 (594h8m14.861272227s) [upload=520837]: Done uploading segments... Dec 30 09:37:14.990207287 (594h8m14.861290322s) [upload=520837]: Upload finished. err: <nil> Duration: 6.105718711s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [X...X.X....X.X...X........X.....X....XXX..X.X.X.......XX....] duration: 3.341952702s trace: https://jaeger.dev.storj.io/trace/957d1f292d8ac53 FAIL (2025-12-30 09:38:06 UTC) orbiter warning: context deadline exceeded Duration: 3.341978973s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..........................X.............................] duration: 29.822185739s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:38:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 29.822202072s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X....X...X..XX..XX......X...X...X...X...X..............X] duration: 30.003201586s trace: https://jaeger.dev.storj.io/trace/1d49fa267a105226 FAIL (2025-12-30 09:37:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: JAGXQBFZAT47VYG6QPABG3VLF5GG5ZXXD7QZQVUUBZXDBOZIUTPQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: FQFAKYDGKF436U6LYDFULMTY5GGHRBBDJWNH2H5GBLOPRPYQRBCA) context deadline exceeded ======LOGS====== Dec 30 09:37:11.998184747 (594h8m11.869267782s) [upload=520837]: Starting upload Dec 30 09:37:11.998242858 (594h8m11.869325903s) [upload=520837]: Done starting upload Dec 30 09:37:11.998320736 (594h8m11.869403781s) [upload=520837]: Uploading segments... Dec 30 09:37:11.998430305 (594h8m11.86951334s) [upload=520837]: Got next segment. Inline: false Dec 30 09:37:11.998497683 (594h8m11.869580718s) [upload=520837]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:37:11.998503394 (594h8m11.869586429s) [upload=520837]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:37:12.245423483 (594h8m12.116506528s) [seg_pos={0 0}, upload=520837]: Begin upload segment... Dec 30 09:37:12.245489709 (594h8m12.116572754s) [seg_pos={0 0}, upload=520837]: Done begin upload segment. Dec 30 09:37:12.245517632 (594h8m12.116600678s) [noise=true , address=145.79.227.44:4195 , piece_id=4NY5NQA4EENHLLNVHSXYF4MND3PXNAZZMQ2BEU5AYPHBMOH52X5Q, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.245528613 (594h8m12.116611658s) [noise=true , address=45.91.116.12:4225 , piece_id=FQFAKYDGKF436U6LYDFULMTY5GGHRBBDJWNH2H5GBLOPRPYQRBCA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.245603135 (594h8m12.11668617s) [noise=true , address=145.79.227.42:4122 , piece_id=D54AIG6TEK6RYR5YRFRV6YYKK6GPNJKM4XXH27JGF2LHEND2EFRA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.246052551 (594h8m12.117135596s) [noise=true , address=145.79.227.47:4265 , piece_id=LYN4UELULFAVN35SSYBFNKNGPMPTPWUK5WTNIE7XAQXIDQNKB4MQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.246425802 (594h8m12.117508837s) [noise=true , address=45.91.116.13:4153 , piece_id=PSBTEW3FXGT7KCV3MOPTJUWOG4OHEAPHW3SL4LFP6R2UJSME2KKA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.246816255 (594h8m12.1178993s) [noise=true , address=145.79.227.25:4617 , piece_id=XHCPRRMPIFB6QUJ7VAGOK7FB57O3PKSXXTDIRQGOPHURYCC22PQA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.247202450 (594h8m12.118285495s) [noise=true , address=145.79.227.7:4294 , piece_id=DJBOJUYN7HQRVSJ2BIACUETF4R3PWZYBLMGR37KC42SGUMII7ENQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.247776564 (594h8m12.118859609s) [noise=true , address=145.79.227.27:4691 , piece_id=JIHOPPJTLAJ57NXWUAMFNXYCUHWRACD3W7AX52V3M5PUBVRNXQLA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.248267669 (594h8m12.119350714s) [noise=true , address=145.79.227.26:4653 , piece_id=THCXKJBYHIBGZVBYGADFTPSF65RAI7TVEXK3W45EW76LAZSTMOQA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.249255339 (594h8m12.120338384s) [noise=true , address=145.79.227.15:4876 , piece_id=LZZTIOEGI27YB2P6IPFFUEBTSMKPCI4QJA6QDAWMDXSBVOOOMHWA, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.249920295 (594h8m12.12100334s) [noise=true , address=45.91.116.11:4117 , piece_id=VQHZGR5NJEMYTKLOJEXL6Q4AMJSIMU3MOULK4OBDAZTPQTJ4ODBQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.250646468 (594h8m12.121729503s) [noise=true , address=145.79.227.45:4231 , piece_id=GNS33UBQPHYRRX2Z5KVU3YHVQYBU6UIJ72FK3KVPVKOQE6FSW6QQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.251624090 (594h8m12.122707135s) [noise=true , address=45.91.116.10:4082 , piece_id=KBAB24TRI3LSPSH6FHG5T2LTSNCVRORMA7DFVKX7KKNNKAKK3INQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.251869908 (594h8m12.122952943s) [noise=true , address=45.91.116.5:4298 , piece_id=JAGXQBFZAT47VYG6QPABG3VLF5GG5ZXXD7QZQVUUBZXDBOZIUTPQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.253106672 (594h8m12.124189717s) [noise=true , address=145.79.227.16:4910 , piece_id=4DFEN2XQUPTBPKSEGQHVESAUKLA6BJBQWWHRM6RQ7BWBT6YJJW7Q, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:12.254630093 (594h8m12.125713128s) [noise=true , address=145.79.227.28:4726 , piece_id=DJWWNLC7KVY7BI6S2D2NHKAOH73SXJX6XO6K72T2DFLMJJ6SUOZQ, seg_pos={0 0}, upload=520837]: Uploading piece... Dec 30 09:37:13.946908938 (594h8m13.817991983s) [upload=520837]: Next returned nil segment Dec 30 09:37:13.946921903 (594h8m13.818004938s) [upload=520837]: Waiting for error group managing segments... Dec 30 09:37:14.227917284 (594h8m14.099000329s) [noise=true , address=145.79.227.16:4910 , piece_id=4DFEN2XQUPTBPKSEGQHVESAUKLA6BJBQWWHRM6RQ7BWBT6YJJW7Q, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.239212040 (594h8m14.110295075s) [noise=true , address=145.79.227.15:4876 , piece_id=LZZTIOEGI27YB2P6IPFFUEBTSMKPCI4QJA6QDAWMDXSBVOOOMHWA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.241175229 (594h8m14.112258264s) [noise=true , address=145.79.227.28:4726 , piece_id=DJWWNLC7KVY7BI6S2D2NHKAOH73SXJX6XO6K72T2DFLMJJ6SUOZQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.243023449 (594h8m14.114106484s) [noise=true , address=145.79.227.42:4122 , piece_id=D54AIG6TEK6RYR5YRFRV6YYKK6GPNJKM4XXH27JGF2LHEND2EFRA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.246043018 (594h8m14.117126053s) [noise=true , address=145.79.227.25:4617 , piece_id=XHCPRRMPIFB6QUJ7VAGOK7FB57O3PKSXXTDIRQGOPHURYCC22PQA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.248630514 (594h8m14.119713549s) [noise=true , address=145.79.227.45:4231 , piece_id=GNS33UBQPHYRRX2Z5KVU3YHVQYBU6UIJ72FK3KVPVKOQE6FSW6QQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.262080175 (594h8m14.13316322s) [noise=true , address=145.79.227.7:4294 , piece_id=DJBOJUYN7HQRVSJ2BIACUETF4R3PWZYBLMGR37KC42SGUMII7ENQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.274266961 (594h8m14.145349996s) [noise=true , address=145.79.227.47:4265 , piece_id=LYN4UELULFAVN35SSYBFNKNGPMPTPWUK5WTNIE7XAQXIDQNKB4MQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.328127120 (594h8m14.199210165s) [noise=true , address=145.79.227.26:4653 , piece_id=THCXKJBYHIBGZVBYGADFTPSF65RAI7TVEXK3W45EW76LAZSTMOQA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.331722776 (594h8m14.202805811s) [noise=true , address=145.79.227.27:4691 , piece_id=JIHOPPJTLAJ57NXWUAMFNXYCUHWRACD3W7AX52V3M5PUBVRNXQLA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.580768008 (594h8m14.451851053s) [noise=true , address=45.91.116.5:4298 , piece_id=JAGXQBFZAT47VYG6QPABG3VLF5GG5ZXXD7QZQVUUBZXDBOZIUTPQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.663882332 (594h8m14.534965367s) [noise=true , address=45.91.116.12:4225 , piece_id=FQFAKYDGKF436U6LYDFULMTY5GGHRBBDJWNH2H5GBLOPRPYQRBCA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.691478029 (594h8m14.562561064s) [noise=true , address=45.91.116.11:4117 , piece_id=VQHZGR5NJEMYTKLOJEXL6Q4AMJSIMU3MOULK4OBDAZTPQTJ4ODBQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: <nil> Dec 30 09:37:14.691496694 (594h8m14.562579729s) [seg_pos={0 0}, upload=520837]: Segment reached optimal threshold of 13 pieces. Dec 30 09:37:14.691839757 (594h8m14.562922802s) [noise=true , address=45.91.116.13:4153 , piece_id=PSBTEW3FXGT7KCV3MOPTJUWOG4OHEAPHW3SL4LFP6R2UJSME2KKA, seg_pos={0 0}, upload=520837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 09:37:14.691849455 (594h8m14.56293249s) [noise=true , address=45.91.116.13:4153 , piece_id=PSBTEW3FXGT7KCV3MOPTJUWOG4OHEAPHW3SL4LFP6R2UJSME2KKA, seg_pos={0 0}, upload=520837]: Long tail cancellation due to optimal threshold Dec 30 09:37:14.691862962 (594h8m14.562945997s) [noise=true , address=45.91.116.10:4082 , piece_id=KBAB24TRI3LSPSH6FHG5T2LTSNCVRORMA7DFVKX7KKNNKAKK3INQ, seg_pos={0 0}, upload=520837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:60096->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:60096->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60096->45.91.116.10:4082: use of closed network connection Dec 30 09:37:14.691874063 (594h8m14.562957098s) [noise=true , address=45.91.116.10:4082 , piece_id=KBAB24TRI3LSPSH6FHG5T2LTSNCVRORMA7DFVKX7KKNNKAKK3INQ, seg_pos={0 0}, upload=520837]: Long tail cancellation due to optimal threshold Dec 30 09:37:14.691990314 (594h8m14.563073349s) [noise=true , address=145.79.227.44:4195 , piece_id=4NY5NQA4EENHLLNVHSXYF4MND3PXNAZZMQ2BEU5AYPHBMOH52X5Q, seg_pos={0 0}, upload=520837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 09:37:14.691999542 (594h8m14.563082587s) [noise=true , address=145.79.227.44:4195 , piece_id=4NY5NQA4EENHLLNVHSXYF4MND3PXNAZZMQ2BEU5AYPHBMOH52X5Q, seg_pos={0 0}, upload=520837]: Long tail cancellation due to optimal threshold Dec 30 09:37:14.692022204 (594h8m14.563105239s) [seg_pos={0 0}, upload=520837]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:37:14.692128346 (594h8m14.563211392s) [upload=520837]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:37:14.692135341 (594h8m14.563218386s) [upload=520837]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:37:14.990189192 (594h8m14.861272227s) [upload=520837]: Done uploading segments... Dec 30 09:37:14.990207287 (594h8m14.861290322s) [upload=520837]: Upload finished. err: <nil> Duration: 30.003226765s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..........................X............................] duration: 36.516071228s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:37:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 36.516086546s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.X...X.X....X.X...X........X.....X....XXX..X.X.X.......XX...] duration: 3.618197857s trace: https://jaeger.dev.storj.io/trace/957d1f292d8ac53 FAIL (2025-12-30 09:37:00 UTC) orbiter warning: context deadline exceeded Duration: 3.618221072s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..........................X...........................] duration: 20.818118256s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:36:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 20.818145769s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X....X...X..XX..XX......X...X...X...X...X..............] duration: 5.870276164s trace: https://jaeger.dev.storj.io/trace/6027170c118f7351 FAIL (2025-12-30 09:36:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: DCYD367QGVNRKJGZV7O7XNODVJNGZRRMBYSYBOYU7U7CTVA4PASQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: ZPGQTJPOEW527ZYVEH3T7NB24DNJNI5TD6MMP7ZUASDTQMAXLQHA) context deadline exceeded ======LOGS====== Dec 30 09:24:55.575505973 (593h55m55.446589018s) [upload=520669]: Starting upload Dec 30 09:24:55.575579052 (593h55m55.446662087s) [upload=520669]: Done starting upload Dec 30 09:24:55.575788261 (593h55m55.446871306s) [upload=520669]: Uploading segments... Dec 30 09:24:55.575924200 (593h55m55.447007245s) [upload=520669]: Got next segment. Inline: false Dec 30 09:24:55.575950589 (593h55m55.447033624s) [upload=520669]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:24:55.575955359 (593h55m55.447038394s) [upload=520669]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:24:55.823145509 (593h55m55.694228544s) [seg_pos={0 0}, upload=520669]: Begin upload segment... Dec 30 09:24:55.823185495 (593h55m55.69426853s) [seg_pos={0 0}, upload=520669]: Done begin upload segment. Dec 30 09:24:55.823209771 (593h55m55.694292806s) [noise=true , address=77.90.31.15:4697 , piece_id=CLZ3LIPPGL24VUZ5PGYAGVNKYD2GY4GTZS6YOK4A4V647X2HYZLQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.823297749 (593h55m55.694380794s) [noise=true , address=45.91.116.5:4299 , piece_id=B4HKNHY4BMEJYWA757C2IE6NFRNHC6LY4POCQX424QDCTDAATGVA, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.823762474 (593h55m55.694845519s) [noise=true , address=145.79.227.23:4547 , piece_id=NXCGOL2NZNGMJF3UJHFZ27YULQGYOQ62BX4QCKUUPCFZX5J2UYTQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.824172234 (593h55m55.69525527s) [noise=true , address=145.79.227.50:4374 , piece_id=UBT3M4UO5RWRANQRJGRRD2ZO47TDI6VJNGP55IR4S3SY43ER7GOQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.824557979 (593h55m55.695641024s) [noise=true , address=145.79.227.29:4942 , piece_id=5LYRE3HLUENAL623FEGT4LAZYIQWLSYWUYJ4DUFL5P5HUEIQCO4Q, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.824570492 (593h55m55.695653537s) [noise=true , address=145.79.227.8:4330 , piece_id=QMCNSXP45AX5KSY2HKATVO326ZXSHF2G4I3JJY7LFI6IIJTHZQHA, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.824869061 (593h55m55.695952106s) [noise=true , address=45.91.116.7:4011 , piece_id=4NLY6SUE53OEKUGQW663ZENOWORH67EJQL2VROXD7JFUTJUT3IYA, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.825036320 (593h55m55.696119355s) [noise=true , address=77.90.31.21:4878 , piece_id=DCYD367QGVNRKJGZV7O7XNODVJNGZRRMBYSYBOYU7U7CTVA4PASQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.825159835 (593h55m55.69624287s) [noise=true , address=145.79.227.20:4438 , piece_id=SWTUJYBB3C7GJPMWNTNK47HFNA635KVNEMONCZQZS2X6FRI37CPQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.825498390 (593h55m55.696581435s) [noise=true , address=45.91.116.4:4939 , piece_id=F6MXQUYFZFLVJCGGDAM4G7LPAECPFPOTLCYYPZNTEYZ5GEYH7A2A, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.825539227 (593h55m55.696622262s) [noise=true , address=45.91.116.10:4081 , piece_id=ZPGQTJPOEW527ZYVEH3T7NB24DNJNI5TD6MMP7ZUASDTQMAXLQHA, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.825900134 (593h55m55.696983179s) [noise=true , address=145.79.227.45:4230 , piece_id=TREJZ7XEZO4DPHAVPHZWMPC3SOESJOOKQE454ZIED4JMB7GVKNLQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.826715567 (593h55m55.697798602s) [noise=true , address=145.79.227.15:4875 , piece_id=IHB7W72DYYJ6CMGZL6L246WKV42555LRKQX7VZUJ2QWCMMZLEUJA, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.827200170 (593h55m55.698283205s) [noise=true , address=145.79.227.32:4836 , piece_id=4WY3XU364LVBHX7OMUZRPXMZIPWCSY6GE6Y2Z5N5LOYAMYXQ5PXQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.827774483 (593h55m55.698857528s) [noise=true , address=145.79.227.41:4087 , piece_id=SPIEPI56NCDRUJBD7T5AB2A6H774N4NK2J6N4JMNQEKEQ5KOVXUQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.836594910 (593h55m55.707677945s) [noise=true , address=145.79.227.25:4619 , piece_id=IBQNYVVSBX27K5FFFALJGLMOKDP4MKCWQJEYSAR2DHAUHZNMUHYQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:57.456610854 (593h55m57.327693899s) [upload=520669]: Next returned nil segment Dec 30 09:24:57.456626703 (593h55m57.327709749s) [upload=520669]: Waiting for error group managing segments... Dec 30 09:24:57.818403545 (593h55m57.68948658s) [noise=true , address=145.79.227.15:4875 , piece_id=IHB7W72DYYJ6CMGZL6L246WKV42555LRKQX7VZUJ2QWCMMZLEUJA, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.818492385 (593h55m57.68957542s) [noise=true , address=145.79.227.32:4836 , piece_id=4WY3XU364LVBHX7OMUZRPXMZIPWCSY6GE6Y2Z5N5LOYAMYXQ5PXQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.820875693 (593h55m57.691958738s) [noise=true , address=145.79.227.45:4230 , piece_id=TREJZ7XEZO4DPHAVPHZWMPC3SOESJOOKQE454ZIED4JMB7GVKNLQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.825019143 (593h55m57.696102178s) [noise=true , address=145.79.227.41:4087 , piece_id=SPIEPI56NCDRUJBD7T5AB2A6H774N4NK2J6N4JMNQEKEQ5KOVXUQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.830504718 (593h55m57.701587764s) [noise=true , address=145.79.227.20:4438 , piece_id=SWTUJYBB3C7GJPMWNTNK47HFNA635KVNEMONCZQZS2X6FRI37CPQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.844793546 (593h55m57.715876591s) [noise=true , address=145.79.227.50:4374 , piece_id=UBT3M4UO5RWRANQRJGRRD2ZO47TDI6VJNGP55IR4S3SY43ER7GOQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.850252141 (593h55m57.721335176s) [noise=true , address=145.79.227.8:4330 , piece_id=QMCNSXP45AX5KSY2HKATVO326ZXSHF2G4I3JJY7LFI6IIJTHZQHA, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.859266415 (593h55m57.730349461s) [noise=true , address=145.79.227.25:4619 , piece_id=IBQNYVVSBX27K5FFFALJGLMOKDP4MKCWQJEYSAR2DHAUHZNMUHYQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.906543097 (593h55m57.777626142s) [noise=true , address=145.79.227.23:4547 , piece_id=NXCGOL2NZNGMJF3UJHFZ27YULQGYOQ62BX4QCKUUPCFZX5J2UYTQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.912673161 (593h55m57.783756196s) [noise=true , address=145.79.227.29:4942 , piece_id=5LYRE3HLUENAL623FEGT4LAZYIQWLSYWUYJ4DUFL5P5HUEIQCO4Q, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:58.097453863 (593h55m57.968536898s) [noise=true , address=77.90.31.21:4878 , piece_id=DCYD367QGVNRKJGZV7O7XNODVJNGZRRMBYSYBOYU7U7CTVA4PASQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:58.127193953 (593h55m57.998276998s) [noise=true , address=45.91.116.10:4081 , piece_id=ZPGQTJPOEW527ZYVEH3T7NB24DNJNI5TD6MMP7ZUASDTQMAXLQHA, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:58.152435096 (593h55m58.023518142s) [noise=true , address=45.91.116.4:4939 , piece_id=F6MXQUYFZFLVJCGGDAM4G7LPAECPFPOTLCYYPZNTEYZ5GEYH7A2A, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:58.152460424 (593h55m58.023543469s) [seg_pos={0 0}, upload=520669]: Segment reached optimal threshold of 13 pieces. Dec 30 09:24:58.152728455 (593h55m58.0238115s) [noise=true , address=77.90.31.15:4697 , piece_id=CLZ3LIPPGL24VUZ5PGYAGVNKYD2GY4GTZS6YOK4A4V647X2HYZLQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:47324->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:47324->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47324->77.90.31.15:4697: use of closed network connection Dec 30 09:24:58.152738113 (593h55m58.023821168s) [noise=true , address=77.90.31.15:4697 , piece_id=CLZ3LIPPGL24VUZ5PGYAGVNKYD2GY4GTZS6YOK4A4V647X2HYZLQ, seg_pos={0 0}, upload=520669]: Long tail cancellation due to optimal threshold Dec 30 09:24:58.152935439 (593h55m58.024018474s) [noise=true , address=45.91.116.7:4011 , piece_id=4NLY6SUE53OEKUGQW663ZENOWORH67EJQL2VROXD7JFUTJUT3IYA, seg_pos={0 0}, upload=520669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 30 09:24:58.152970486 (593h55m58.024053531s) [noise=true , address=45.91.116.7:4011 , piece_id=4NLY6SUE53OEKUGQW663ZENOWORH67EJQL2VROXD7JFUTJUT3IYA, seg_pos={0 0}, upload=520669]: Long tail cancellation due to optimal threshold Dec 30 09:24:58.153192618 (593h55m58.024275663s) [noise=true , address=45.91.116.5:4299 , piece_id=B4HKNHY4BMEJYWA757C2IE6NFRNHC6LY4POCQX424QDCTDAATGVA, seg_pos={0 0}, upload=520669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 09:24:58.153201325 (593h55m58.024284361s) [noise=true , address=45.91.116.5:4299 , piece_id=B4HKNHY4BMEJYWA757C2IE6NFRNHC6LY4POCQX424QDCTDAATGVA, seg_pos={0 0}, upload=520669]: Long tail cancellation due to optimal threshold Dec 30 09:24:58.153237685 (593h55m58.02432073s) [seg_pos={0 0}, upload=520669]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:24:58.153353605 (593h55m58.02443665s) [upload=520669]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:24:58.153361149 (593h55m58.024444194s) [upload=520669]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:24:58.440093234 (593h55m58.311176269s) [upload=520669]: Done uploading segments... Dec 30 09:24:58.440126056 (593h55m58.311209101s) [upload=520669]: Upload finished. err: <nil> Duration: 5.870301252s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [..X...X.X....X.X...X........X.....X....XXX..X.X.X.......XX..] duration: 3.56455916s trace: https://jaeger.dev.storj.io/trace/957d1f292d8ac53 FAIL (2025-12-30 09:36:14 UTC) orbiter warning: context deadline exceeded Duration: 3.564603526s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..........................X..........................] duration: 37.732934227s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:36:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 37.733012566s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [...X...X.X....X.X...X........X.....X....XXX..X.X.X.......XX.] duration: 5.001207562s trace: https://jaeger.dev.storj.io/trace/957d1f292d8ac53 FAIL (2025-12-30 09:35:51 UTC) orbiter warning: context deadline exceeded Duration: 5.001229374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X....X...X..XX..XX......X...X...X...X...X.............] duration: 5.749118877s trace: https://jaeger.dev.storj.io/trace/6027170c118f7351 FAIL (2025-12-30 09:35:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: DCYD367QGVNRKJGZV7O7XNODVJNGZRRMBYSYBOYU7U7CTVA4PASQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: ZPGQTJPOEW527ZYVEH3T7NB24DNJNI5TD6MMP7ZUASDTQMAXLQHA) context deadline exceeded ======LOGS====== Dec 30 09:24:55.575505973 (593h55m55.446589018s) [upload=520669]: Starting upload Dec 30 09:24:55.575579052 (593h55m55.446662087s) [upload=520669]: Done starting upload Dec 30 09:24:55.575788261 (593h55m55.446871306s) [upload=520669]: Uploading segments... Dec 30 09:24:55.575924200 (593h55m55.447007245s) [upload=520669]: Got next segment. Inline: false Dec 30 09:24:55.575950589 (593h55m55.447033624s) [upload=520669]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:24:55.575955359 (593h55m55.447038394s) [upload=520669]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:24:55.823145509 (593h55m55.694228544s) [seg_pos={0 0}, upload=520669]: Begin upload segment... Dec 30 09:24:55.823185495 (593h55m55.69426853s) [seg_pos={0 0}, upload=520669]: Done begin upload segment. Dec 30 09:24:55.823209771 (593h55m55.694292806s) [noise=true , address=77.90.31.15:4697 , piece_id=CLZ3LIPPGL24VUZ5PGYAGVNKYD2GY4GTZS6YOK4A4V647X2HYZLQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.823297749 (593h55m55.694380794s) [noise=true , address=45.91.116.5:4299 , piece_id=B4HKNHY4BMEJYWA757C2IE6NFRNHC6LY4POCQX424QDCTDAATGVA, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.823762474 (593h55m55.694845519s) [noise=true , address=145.79.227.23:4547 , piece_id=NXCGOL2NZNGMJF3UJHFZ27YULQGYOQ62BX4QCKUUPCFZX5J2UYTQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.824172234 (593h55m55.69525527s) [noise=true , address=145.79.227.50:4374 , piece_id=UBT3M4UO5RWRANQRJGRRD2ZO47TDI6VJNGP55IR4S3SY43ER7GOQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.824557979 (593h55m55.695641024s) [noise=true , address=145.79.227.29:4942 , piece_id=5LYRE3HLUENAL623FEGT4LAZYIQWLSYWUYJ4DUFL5P5HUEIQCO4Q, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.824570492 (593h55m55.695653537s) [noise=true , address=145.79.227.8:4330 , piece_id=QMCNSXP45AX5KSY2HKATVO326ZXSHF2G4I3JJY7LFI6IIJTHZQHA, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.824869061 (593h55m55.695952106s) [noise=true , address=45.91.116.7:4011 , piece_id=4NLY6SUE53OEKUGQW663ZENOWORH67EJQL2VROXD7JFUTJUT3IYA, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.825036320 (593h55m55.696119355s) [noise=true , address=77.90.31.21:4878 , piece_id=DCYD367QGVNRKJGZV7O7XNODVJNGZRRMBYSYBOYU7U7CTVA4PASQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.825159835 (593h55m55.69624287s) [noise=true , address=145.79.227.20:4438 , piece_id=SWTUJYBB3C7GJPMWNTNK47HFNA635KVNEMONCZQZS2X6FRI37CPQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.825498390 (593h55m55.696581435s) [noise=true , address=45.91.116.4:4939 , piece_id=F6MXQUYFZFLVJCGGDAM4G7LPAECPFPOTLCYYPZNTEYZ5GEYH7A2A, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.825539227 (593h55m55.696622262s) [noise=true , address=45.91.116.10:4081 , piece_id=ZPGQTJPOEW527ZYVEH3T7NB24DNJNI5TD6MMP7ZUASDTQMAXLQHA, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.825900134 (593h55m55.696983179s) [noise=true , address=145.79.227.45:4230 , piece_id=TREJZ7XEZO4DPHAVPHZWMPC3SOESJOOKQE454ZIED4JMB7GVKNLQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.826715567 (593h55m55.697798602s) [noise=true , address=145.79.227.15:4875 , piece_id=IHB7W72DYYJ6CMGZL6L246WKV42555LRKQX7VZUJ2QWCMMZLEUJA, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.827200170 (593h55m55.698283205s) [noise=true , address=145.79.227.32:4836 , piece_id=4WY3XU364LVBHX7OMUZRPXMZIPWCSY6GE6Y2Z5N5LOYAMYXQ5PXQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.827774483 (593h55m55.698857528s) [noise=true , address=145.79.227.41:4087 , piece_id=SPIEPI56NCDRUJBD7T5AB2A6H774N4NK2J6N4JMNQEKEQ5KOVXUQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.836594910 (593h55m55.707677945s) [noise=true , address=145.79.227.25:4619 , piece_id=IBQNYVVSBX27K5FFFALJGLMOKDP4MKCWQJEYSAR2DHAUHZNMUHYQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:57.456610854 (593h55m57.327693899s) [upload=520669]: Next returned nil segment Dec 30 09:24:57.456626703 (593h55m57.327709749s) [upload=520669]: Waiting for error group managing segments... Dec 30 09:24:57.818403545 (593h55m57.68948658s) [noise=true , address=145.79.227.15:4875 , piece_id=IHB7W72DYYJ6CMGZL6L246WKV42555LRKQX7VZUJ2QWCMMZLEUJA, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.818492385 (593h55m57.68957542s) [noise=true , address=145.79.227.32:4836 , piece_id=4WY3XU364LVBHX7OMUZRPXMZIPWCSY6GE6Y2Z5N5LOYAMYXQ5PXQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.820875693 (593h55m57.691958738s) [noise=true , address=145.79.227.45:4230 , piece_id=TREJZ7XEZO4DPHAVPHZWMPC3SOESJOOKQE454ZIED4JMB7GVKNLQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.825019143 (593h55m57.696102178s) [noise=true , address=145.79.227.41:4087 , piece_id=SPIEPI56NCDRUJBD7T5AB2A6H774N4NK2J6N4JMNQEKEQ5KOVXUQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.830504718 (593h55m57.701587764s) [noise=true , address=145.79.227.20:4438 , piece_id=SWTUJYBB3C7GJPMWNTNK47HFNA635KVNEMONCZQZS2X6FRI37CPQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.844793546 (593h55m57.715876591s) [noise=true , address=145.79.227.50:4374 , piece_id=UBT3M4UO5RWRANQRJGRRD2ZO47TDI6VJNGP55IR4S3SY43ER7GOQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.850252141 (593h55m57.721335176s) [noise=true , address=145.79.227.8:4330 , piece_id=QMCNSXP45AX5KSY2HKATVO326ZXSHF2G4I3JJY7LFI6IIJTHZQHA, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.859266415 (593h55m57.730349461s) [noise=true , address=145.79.227.25:4619 , piece_id=IBQNYVVSBX27K5FFFALJGLMOKDP4MKCWQJEYSAR2DHAUHZNMUHYQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.906543097 (593h55m57.777626142s) [noise=true , address=145.79.227.23:4547 , piece_id=NXCGOL2NZNGMJF3UJHFZ27YULQGYOQ62BX4QCKUUPCFZX5J2UYTQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.912673161 (593h55m57.783756196s) [noise=true , address=145.79.227.29:4942 , piece_id=5LYRE3HLUENAL623FEGT4LAZYIQWLSYWUYJ4DUFL5P5HUEIQCO4Q, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:58.097453863 (593h55m57.968536898s) [noise=true , address=77.90.31.21:4878 , piece_id=DCYD367QGVNRKJGZV7O7XNODVJNGZRRMBYSYBOYU7U7CTVA4PASQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:58.127193953 (593h55m57.998276998s) [noise=true , address=45.91.116.10:4081 , piece_id=ZPGQTJPOEW527ZYVEH3T7NB24DNJNI5TD6MMP7ZUASDTQMAXLQHA, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:58.152435096 (593h55m58.023518142s) [noise=true , address=45.91.116.4:4939 , piece_id=F6MXQUYFZFLVJCGGDAM4G7LPAECPFPOTLCYYPZNTEYZ5GEYH7A2A, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:58.152460424 (593h55m58.023543469s) [seg_pos={0 0}, upload=520669]: Segment reached optimal threshold of 13 pieces. Dec 30 09:24:58.152728455 (593h55m58.0238115s) [noise=true , address=77.90.31.15:4697 , piece_id=CLZ3LIPPGL24VUZ5PGYAGVNKYD2GY4GTZS6YOK4A4V647X2HYZLQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:47324->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:47324->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47324->77.90.31.15:4697: use of closed network connection Dec 30 09:24:58.152738113 (593h55m58.023821168s) [noise=true , address=77.90.31.15:4697 , piece_id=CLZ3LIPPGL24VUZ5PGYAGVNKYD2GY4GTZS6YOK4A4V647X2HYZLQ, seg_pos={0 0}, upload=520669]: Long tail cancellation due to optimal threshold Dec 30 09:24:58.152935439 (593h55m58.024018474s) [noise=true , address=45.91.116.7:4011 , piece_id=4NLY6SUE53OEKUGQW663ZENOWORH67EJQL2VROXD7JFUTJUT3IYA, seg_pos={0 0}, upload=520669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 30 09:24:58.152970486 (593h55m58.024053531s) [noise=true , address=45.91.116.7:4011 , piece_id=4NLY6SUE53OEKUGQW663ZENOWORH67EJQL2VROXD7JFUTJUT3IYA, seg_pos={0 0}, upload=520669]: Long tail cancellation due to optimal threshold Dec 30 09:24:58.153192618 (593h55m58.024275663s) [noise=true , address=45.91.116.5:4299 , piece_id=B4HKNHY4BMEJYWA757C2IE6NFRNHC6LY4POCQX424QDCTDAATGVA, seg_pos={0 0}, upload=520669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 09:24:58.153201325 (593h55m58.024284361s) [noise=true , address=45.91.116.5:4299 , piece_id=B4HKNHY4BMEJYWA757C2IE6NFRNHC6LY4POCQX424QDCTDAATGVA, seg_pos={0 0}, upload=520669]: Long tail cancellation due to optimal threshold Dec 30 09:24:58.153237685 (593h55m58.02432073s) [seg_pos={0 0}, upload=520669]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:24:58.153353605 (593h55m58.02443665s) [upload=520669]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:24:58.153361149 (593h55m58.024444194s) [upload=520669]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:24:58.440093234 (593h55m58.311176269s) [upload=520669]: Done uploading segments... Dec 30 09:24:58.440126056 (593h55m58.311209101s) [upload=520669]: Upload finished. err: <nil> Duration: 5.749177979s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..........................X.........................] duration: 19.903929797s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:35:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 19.903944766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X......X....X...X..XX..XX......X...X...X...X...X............] duration: 5.982200591s trace: https://jaeger.dev.storj.io/trace/6027170c118f7351 FAIL (2025-12-30 09:35:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: DCYD367QGVNRKJGZV7O7XNODVJNGZRRMBYSYBOYU7U7CTVA4PASQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: ZPGQTJPOEW527ZYVEH3T7NB24DNJNI5TD6MMP7ZUASDTQMAXLQHA) context deadline exceeded ======LOGS====== Dec 30 09:24:55.575505973 (593h55m55.446589018s) [upload=520669]: Starting upload Dec 30 09:24:55.575579052 (593h55m55.446662087s) [upload=520669]: Done starting upload Dec 30 09:24:55.575788261 (593h55m55.446871306s) [upload=520669]: Uploading segments... Dec 30 09:24:55.575924200 (593h55m55.447007245s) [upload=520669]: Got next segment. Inline: false Dec 30 09:24:55.575950589 (593h55m55.447033624s) [upload=520669]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:24:55.575955359 (593h55m55.447038394s) [upload=520669]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:24:55.823145509 (593h55m55.694228544s) [seg_pos={0 0}, upload=520669]: Begin upload segment... Dec 30 09:24:55.823185495 (593h55m55.69426853s) [seg_pos={0 0}, upload=520669]: Done begin upload segment. Dec 30 09:24:55.823209771 (593h55m55.694292806s) [noise=true , address=77.90.31.15:4697 , piece_id=CLZ3LIPPGL24VUZ5PGYAGVNKYD2GY4GTZS6YOK4A4V647X2HYZLQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.823297749 (593h55m55.694380794s) [noise=true , address=45.91.116.5:4299 , piece_id=B4HKNHY4BMEJYWA757C2IE6NFRNHC6LY4POCQX424QDCTDAATGVA, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.823762474 (593h55m55.694845519s) [noise=true , address=145.79.227.23:4547 , piece_id=NXCGOL2NZNGMJF3UJHFZ27YULQGYOQ62BX4QCKUUPCFZX5J2UYTQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.824172234 (593h55m55.69525527s) [noise=true , address=145.79.227.50:4374 , piece_id=UBT3M4UO5RWRANQRJGRRD2ZO47TDI6VJNGP55IR4S3SY43ER7GOQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.824557979 (593h55m55.695641024s) [noise=true , address=145.79.227.29:4942 , piece_id=5LYRE3HLUENAL623FEGT4LAZYIQWLSYWUYJ4DUFL5P5HUEIQCO4Q, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.824570492 (593h55m55.695653537s) [noise=true , address=145.79.227.8:4330 , piece_id=QMCNSXP45AX5KSY2HKATVO326ZXSHF2G4I3JJY7LFI6IIJTHZQHA, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.824869061 (593h55m55.695952106s) [noise=true , address=45.91.116.7:4011 , piece_id=4NLY6SUE53OEKUGQW663ZENOWORH67EJQL2VROXD7JFUTJUT3IYA, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.825036320 (593h55m55.696119355s) [noise=true , address=77.90.31.21:4878 , piece_id=DCYD367QGVNRKJGZV7O7XNODVJNGZRRMBYSYBOYU7U7CTVA4PASQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.825159835 (593h55m55.69624287s) [noise=true , address=145.79.227.20:4438 , piece_id=SWTUJYBB3C7GJPMWNTNK47HFNA635KVNEMONCZQZS2X6FRI37CPQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.825498390 (593h55m55.696581435s) [noise=true , address=45.91.116.4:4939 , piece_id=F6MXQUYFZFLVJCGGDAM4G7LPAECPFPOTLCYYPZNTEYZ5GEYH7A2A, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.825539227 (593h55m55.696622262s) [noise=true , address=45.91.116.10:4081 , piece_id=ZPGQTJPOEW527ZYVEH3T7NB24DNJNI5TD6MMP7ZUASDTQMAXLQHA, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.825900134 (593h55m55.696983179s) [noise=true , address=145.79.227.45:4230 , piece_id=TREJZ7XEZO4DPHAVPHZWMPC3SOESJOOKQE454ZIED4JMB7GVKNLQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.826715567 (593h55m55.697798602s) [noise=true , address=145.79.227.15:4875 , piece_id=IHB7W72DYYJ6CMGZL6L246WKV42555LRKQX7VZUJ2QWCMMZLEUJA, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.827200170 (593h55m55.698283205s) [noise=true , address=145.79.227.32:4836 , piece_id=4WY3XU364LVBHX7OMUZRPXMZIPWCSY6GE6Y2Z5N5LOYAMYXQ5PXQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.827774483 (593h55m55.698857528s) [noise=true , address=145.79.227.41:4087 , piece_id=SPIEPI56NCDRUJBD7T5AB2A6H774N4NK2J6N4JMNQEKEQ5KOVXUQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:55.836594910 (593h55m55.707677945s) [noise=true , address=145.79.227.25:4619 , piece_id=IBQNYVVSBX27K5FFFALJGLMOKDP4MKCWQJEYSAR2DHAUHZNMUHYQ, seg_pos={0 0}, upload=520669]: Uploading piece... Dec 30 09:24:57.456610854 (593h55m57.327693899s) [upload=520669]: Next returned nil segment Dec 30 09:24:57.456626703 (593h55m57.327709749s) [upload=520669]: Waiting for error group managing segments... Dec 30 09:24:57.818403545 (593h55m57.68948658s) [noise=true , address=145.79.227.15:4875 , piece_id=IHB7W72DYYJ6CMGZL6L246WKV42555LRKQX7VZUJ2QWCMMZLEUJA, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.818492385 (593h55m57.68957542s) [noise=true , address=145.79.227.32:4836 , piece_id=4WY3XU364LVBHX7OMUZRPXMZIPWCSY6GE6Y2Z5N5LOYAMYXQ5PXQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.820875693 (593h55m57.691958738s) [noise=true , address=145.79.227.45:4230 , piece_id=TREJZ7XEZO4DPHAVPHZWMPC3SOESJOOKQE454ZIED4JMB7GVKNLQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.825019143 (593h55m57.696102178s) [noise=true , address=145.79.227.41:4087 , piece_id=SPIEPI56NCDRUJBD7T5AB2A6H774N4NK2J6N4JMNQEKEQ5KOVXUQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.830504718 (593h55m57.701587764s) [noise=true , address=145.79.227.20:4438 , piece_id=SWTUJYBB3C7GJPMWNTNK47HFNA635KVNEMONCZQZS2X6FRI37CPQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.844793546 (593h55m57.715876591s) [noise=true , address=145.79.227.50:4374 , piece_id=UBT3M4UO5RWRANQRJGRRD2ZO47TDI6VJNGP55IR4S3SY43ER7GOQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.850252141 (593h55m57.721335176s) [noise=true , address=145.79.227.8:4330 , piece_id=QMCNSXP45AX5KSY2HKATVO326ZXSHF2G4I3JJY7LFI6IIJTHZQHA, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.859266415 (593h55m57.730349461s) [noise=true , address=145.79.227.25:4619 , piece_id=IBQNYVVSBX27K5FFFALJGLMOKDP4MKCWQJEYSAR2DHAUHZNMUHYQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.906543097 (593h55m57.777626142s) [noise=true , address=145.79.227.23:4547 , piece_id=NXCGOL2NZNGMJF3UJHFZ27YULQGYOQ62BX4QCKUUPCFZX5J2UYTQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:57.912673161 (593h55m57.783756196s) [noise=true , address=145.79.227.29:4942 , piece_id=5LYRE3HLUENAL623FEGT4LAZYIQWLSYWUYJ4DUFL5P5HUEIQCO4Q, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:58.097453863 (593h55m57.968536898s) [noise=true , address=77.90.31.21:4878 , piece_id=DCYD367QGVNRKJGZV7O7XNODVJNGZRRMBYSYBOYU7U7CTVA4PASQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:58.127193953 (593h55m57.998276998s) [noise=true , address=45.91.116.10:4081 , piece_id=ZPGQTJPOEW527ZYVEH3T7NB24DNJNI5TD6MMP7ZUASDTQMAXLQHA, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:58.152435096 (593h55m58.023518142s) [noise=true , address=45.91.116.4:4939 , piece_id=F6MXQUYFZFLVJCGGDAM4G7LPAECPFPOTLCYYPZNTEYZ5GEYH7A2A, seg_pos={0 0}, upload=520669]: Done uploading piece. err: <nil> Dec 30 09:24:58.152460424 (593h55m58.023543469s) [seg_pos={0 0}, upload=520669]: Segment reached optimal threshold of 13 pieces. Dec 30 09:24:58.152728455 (593h55m58.0238115s) [noise=true , address=77.90.31.15:4697 , piece_id=CLZ3LIPPGL24VUZ5PGYAGVNKYD2GY4GTZS6YOK4A4V647X2HYZLQ, seg_pos={0 0}, upload=520669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:47324->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:47324->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47324->77.90.31.15:4697: use of closed network connection Dec 30 09:24:58.152738113 (593h55m58.023821168s) [noise=true , address=77.90.31.15:4697 , piece_id=CLZ3LIPPGL24VUZ5PGYAGVNKYD2GY4GTZS6YOK4A4V647X2HYZLQ, seg_pos={0 0}, upload=520669]: Long tail cancellation due to optimal threshold Dec 30 09:24:58.152935439 (593h55m58.024018474s) [noise=true , address=45.91.116.7:4011 , piece_id=4NLY6SUE53OEKUGQW663ZENOWORH67EJQL2VROXD7JFUTJUT3IYA, seg_pos={0 0}, upload=520669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 30 09:24:58.152970486 (593h55m58.024053531s) [noise=true , address=45.91.116.7:4011 , piece_id=4NLY6SUE53OEKUGQW663ZENOWORH67EJQL2VROXD7JFUTJUT3IYA, seg_pos={0 0}, upload=520669]: Long tail cancellation due to optimal threshold Dec 30 09:24:58.153192618 (593h55m58.024275663s) [noise=true , address=45.91.116.5:4299 , piece_id=B4HKNHY4BMEJYWA757C2IE6NFRNHC6LY4POCQX424QDCTDAATGVA, seg_pos={0 0}, upload=520669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 09:24:58.153201325 (593h55m58.024284361s) [noise=true , address=45.91.116.5:4299 , piece_id=B4HKNHY4BMEJYWA757C2IE6NFRNHC6LY4POCQX424QDCTDAATGVA, seg_pos={0 0}, upload=520669]: Long tail cancellation due to optimal threshold Dec 30 09:24:58.153237685 (593h55m58.02432073s) [seg_pos={0 0}, upload=520669]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:24:58.153353605 (593h55m58.02443665s) [upload=520669]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:24:58.153361149 (593h55m58.024444194s) [upload=520669]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:24:58.440093234 (593h55m58.311176269s) [upload=520669]: Done uploading segments... Dec 30 09:24:58.440126056 (593h55m58.311209101s) [upload=520669]: Upload finished. err: <nil> Duration: 5.982216062s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [....X...X.X....X.X...X........X.....X....XXX..X.X.X.......XX] duration: 5.000694998s trace: https://jaeger.dev.storj.io/trace/957d1f292d8ac53 FAIL (2025-12-30 09:35:10 UTC) orbiter warning: context deadline exceeded Duration: 5.000721609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..........................X........................] duration: 48.408695432s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:35:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 48.408730458s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.....X...X.X....X.X...X........X.....X....XXX..X.X.X.......X] duration: 5.000757546s trace: https://jaeger.dev.storj.io/trace/2947772e0087d791 FAIL (2025-12-30 09:34:27 UTC) orbiter warning: context deadline exceeded Duration: 5.00079085s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)