Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.460477165s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:29:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.460506622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............................................X........XX.X...] duration: 3.548427941s trace: https://jaeger.dev.storj.io/trace/65f995f41356343d FAIL (2025-12-25 12:29:24 UTC) orbiter warning: context deadline exceeded Duration: 3.548455504s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............................................X........XX.X..] duration: 3.523055138s trace: https://jaeger.dev.storj.io/trace/65f995f41356343d FAIL (2025-12-25 12:28:54 UTC) orbiter warning: context deadline exceeded Duration: 3.523091145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.361352739s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:28:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.361367788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............................................X........XX.X.] duration: 3.579021248s trace: https://jaeger.dev.storj.io/trace/65f995f41356343d FAIL (2025-12-25 12:28:18 UTC) orbiter warning: context deadline exceeded Duration: 3.579040884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.664136681s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:28:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.664163361s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...............................................X........XX.X] duration: 5.001187355s trace: https://jaeger.dev.storj.io/trace/65f995f41356343d FAIL (2025-12-25 12:27:36 UTC) orbiter warning: context deadline exceeded Duration: 5.00120608s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.359220137s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:27:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.35924777s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X...............................................X........XX.] duration: 4.884222938s trace: https://jaeger.dev.storj.io/trace/50ca9251157a60ed FAIL (2025-12-25 12:27:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.884252704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.469488069s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:26:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.469511994s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X...............................................X........XX] duration: 5.001698889s trace: https://jaeger.dev.storj.io/trace/50ca9251157a60ed FAIL (2025-12-25 12:26:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001721472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.374174535s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:26:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.374197229s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X...............................................X........X] duration: 5.001761196s trace: https://jaeger.dev.storj.io/trace/74ab99d10c3e98f0 FAIL (2025-12-25 12:26:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001784181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.575308909s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:25:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.575335611s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X...............................................X........] duration: 4.012870674s trace: https://jaeger.dev.storj.io/trace/c183c6332a8c273 FAIL (2025-12-25 12:25:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.012900289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.663093464s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:25:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.663125625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X...............................................X.......] duration: 3.510475172s trace: https://jaeger.dev.storj.io/trace/c183c6332a8c273 FAIL (2025-12-25 12:24:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.510499138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.480190383s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:24:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 6.480215309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.758514211s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:23:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.758542013s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X...............................................X......] duration: 5.001281044s trace: https://jaeger.dev.storj.io/trace/c183c6332a8c273 FAIL (2025-12-25 12:23:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00131011s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......X...............................................X.....] duration: 4.96148901s trace: https://jaeger.dev.storj.io/trace/c183c6332a8c273 FAIL (2025-12-25 12:23:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.961521351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.766277324s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:22:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.766302332s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.622101257s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:22:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.62212947s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......X...............................................X....] duration: 4.038611381s trace: https://jaeger.dev.storj.io/trace/c183c6332a8c273 FAIL (2025-12-25 12:22:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.038640747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.915192314s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:21:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.915220496s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........X...............................................X...] duration: 5.000675009s trace: https://jaeger.dev.storj.io/trace/c183c6332a8c273 FAIL (2025-12-25 12:21:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000700479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.35101157s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:20:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.351059721s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........X...............................................X..] duration: 3.542441722s trace: https://jaeger.dev.storj.io/trace/c183c6332a8c273 FAIL (2025-12-25 12:20:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.542473943s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........X...............................................X.] duration: 3.514376108s trace: https://jaeger.dev.storj.io/trace/c183c6332a8c273 FAIL (2025-12-25 12:20:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.514399593s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.612565222s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:19:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.612595509s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........X...............................................X] duration: 5.001362178s trace: https://jaeger.dev.storj.io/trace/c183c6332a8c273 FAIL (2025-12-25 12:19:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001388227s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 5.00059157s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:18:52 UTC) orbiter warning: context deadline exceeded Duration: 5.000623983s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.039262874s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:18:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 6.039286018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.384897061s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:18:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 6.384938861s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.488367452s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:18:06 UTC) orbiter warning: context deadline exceeded Duration: 3.488399203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.644876916s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:18:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.644893608s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.581242915s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:17:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.581268083s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.556093134s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:17:19 UTC) orbiter warning: context deadline exceeded Duration: 3.556124724s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.478927689s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:16:55 UTC) orbiter warning: context deadline exceeded Duration: 3.478956014s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.518577993s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:16:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.518604103s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.392889197s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:16:36 UTC) orbiter warning: context deadline exceeded Duration: 3.392921851s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.482899731s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:15:48 UTC) orbiter warning: context deadline exceeded Duration: 3.482926582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.636261625s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:15:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.636277274s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 5.000911783s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:15:30 UTC) orbiter warning: context deadline exceeded Duration: 5.000933705s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 5.000519954s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:15:12 UTC) orbiter warning: context deadline exceeded Duration: 5.000631497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.585004038s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:15:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.585038734s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 5.000925747s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:14:31 UTC) orbiter warning: context deadline exceeded Duration: 5.000949183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.520727858s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:14:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.520824693s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.462488437s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:13:57 UTC) orbiter warning: context deadline exceeded Duration: 3.46251629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.450658212s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:13:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.450687136s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)