Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.........................X........XX.X....X....X.XX...XXXXXX] duration: 5.000990439s trace: https://jaeger.dev.storj.io/trace/7f89a5f0e1e5e9d9 FAIL (2025-12-25 12:43:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001009284s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..........................X........XX.X....X....X.XX...XXXXX] duration: 5.001471364s trace: https://jaeger.dev.storj.io/trace/10bf9b527e233fbc FAIL (2025-12-25 12:43:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001538922s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...........................X........XX.X....X....X.XX...XXXX] duration: 5.002040629s trace: https://jaeger.dev.storj.io/trace/785904e571cce3f6 FAIL (2025-12-25 12:42:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002061569s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [............................X........XX.X....X....X.XX...XXX] duration: 5.001070843s trace: https://jaeger.dev.storj.io/trace/4d1ad37aa26b6126 FAIL (2025-12-25 12:41:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00109549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.4450596s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:40:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.445084888s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.............................X........XX.X....X....X.XX...XX] duration: 5.001059319s trace: https://jaeger.dev.storj.io/trace/813426f4ba5dee3 FAIL (2025-12-25 12:40:41 UTC) orbiter warning: context deadline exceeded Duration: 5.001079497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.503683661s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:40:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.50371422s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..............................X........XX.X....X....X.XX...X] duration: 5.001275293s trace: https://jaeger.dev.storj.io/trace/64c8075328cd1a4f FAIL (2025-12-25 12:40:12 UTC) orbiter warning: context deadline exceeded Duration: 5.001300431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.336426977s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:39:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.336454159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.856144364s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:39:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.856174502s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...............................X........XX.X....X....X.XX...] duration: 5.000771576s trace: https://jaeger.dev.storj.io/trace/7b14c98be68176ca FAIL (2025-12-25 12:39:12 UTC) orbiter warning: context deadline exceeded Duration: 5.000795551s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [................................X........XX.X....X....X.XX..] duration: 3.562732706s trace: https://jaeger.dev.storj.io/trace/7b14c98be68176ca FAIL (2025-12-25 12:38:22 UTC) orbiter warning: context deadline exceeded Duration: 3.562754157s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.549456841s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:38:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.54951901s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.242501333s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:37:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 6.242557009s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.................................X........XX.X....X....X.XX.] duration: 3.614519001s trace: https://jaeger.dev.storj.io/trace/7b14c98be68176ca FAIL (2025-12-25 12:37:34 UTC) orbiter warning: context deadline exceeded Duration: 3.614546492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.543531417s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:37:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.543558589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.545313462s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:36:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.545330344s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..................................X........XX.X....X....X.XX] duration: 5.001171045s trace: https://jaeger.dev.storj.io/trace/7b14c98be68176ca FAIL (2025-12-25 12:36:43 UTC) orbiter warning: context deadline exceeded Duration: 5.001195913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.807079822s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:36:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.807106161s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...................................X........XX.X....X....X.X] duration: 5.001803307s trace: https://jaeger.dev.storj.io/trace/17015cf567a86cf3 FAIL (2025-12-25 12:36:15 UTC) orbiter warning: context deadline exceeded Duration: 5.001825479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.584191894s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:35:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.584224696s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....................................X........XX.X....X....X.] duration: 3.400098808s trace: https://jaeger.dev.storj.io/trace/325e1b1a3935fe2a FAIL (2025-12-25 12:35:06 UTC) orbiter warning: context deadline exceeded Duration: 3.400191294s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....................................X........XX.X....X....X] duration: 5.001503427s trace: https://jaeger.dev.storj.io/trace/325e1b1a3935fe2a FAIL (2025-12-25 12:34:51 UTC) orbiter warning: context deadline exceeded Duration: 5.001528635s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [......................................X........XX.X....X....] duration: 4.525562284s trace: https://jaeger.dev.storj.io/trace/7709d4a7364cf27f FAIL (2025-12-25 12:34:04 UTC) orbiter warning: context deadline exceeded Duration: 4.525589796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.284400369s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:34:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.284430216s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.......................................X........XX.X....X...] duration: 4.593902736s trace: https://jaeger.dev.storj.io/trace/7709d4a7364cf27f FAIL (2025-12-25 12:33:39 UTC) orbiter warning: context deadline exceeded Duration: 4.593978829s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.416986877s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:32:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.417012947s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [........................................X........XX.X....X..] duration: 3.677083805s trace: https://jaeger.dev.storj.io/trace/7709d4a7364cf27f FAIL (2025-12-25 12:32:46 UTC) orbiter warning: context deadline exceeded Duration: 3.677097702s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.........................................X........XX.X....X.] duration: 3.490548029s trace: https://jaeger.dev.storj.io/trace/7709d4a7364cf27f FAIL (2025-12-25 12:32:01 UTC) orbiter warning: context deadline exceeded Duration: 3.490644102s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.402623252s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:31:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 6.402650263s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..........................................X........XX.X....X] duration: 5.001277469s trace: https://jaeger.dev.storj.io/trace/7709d4a7364cf27f FAIL (2025-12-25 12:31:42 UTC) orbiter warning: context deadline exceeded Duration: 5.001305032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.260445993s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:30:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 6.260492302s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........................................X........XX.X....] duration: 3.857235475s trace: https://jaeger.dev.storj.io/trace/65f995f41356343d FAIL (2025-12-25 12:30:39 UTC) orbiter warning: context deadline exceeded Duration: 3.85726502s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.648378685s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:30:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.648409574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.460477165s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:29:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.460506622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............................................X........XX.X...] duration: 3.548427941s trace: https://jaeger.dev.storj.io/trace/65f995f41356343d FAIL (2025-12-25 12:29:24 UTC) orbiter warning: context deadline exceeded Duration: 3.548455504s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............................................X........XX.X..] duration: 3.523055138s trace: https://jaeger.dev.storj.io/trace/65f995f41356343d FAIL (2025-12-25 12:28:54 UTC) orbiter warning: context deadline exceeded Duration: 3.523091145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.361352739s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:28:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.361367788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............................................X........XX.X.] duration: 3.579021248s trace: https://jaeger.dev.storj.io/trace/65f995f41356343d FAIL (2025-12-25 12:28:18 UTC) orbiter warning: context deadline exceeded Duration: 3.579040884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.664136681s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:28:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.664163361s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...............................................X........XX.X] duration: 5.001187355s trace: https://jaeger.dev.storj.io/trace/65f995f41356343d FAIL (2025-12-25 12:27:36 UTC) orbiter warning: context deadline exceeded Duration: 5.00120608s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.359220137s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:27:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.35924777s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X...............................................X........XX.] duration: 4.884222938s trace: https://jaeger.dev.storj.io/trace/50ca9251157a60ed FAIL (2025-12-25 12:27:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.884252704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.469488069s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:26:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.469511994s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X...............................................X........XX] duration: 5.001698889s trace: https://jaeger.dev.storj.io/trace/50ca9251157a60ed FAIL (2025-12-25 12:26:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001721472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.374174535s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:26:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.374197229s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X...............................................X........X] duration: 5.001761196s trace: https://jaeger.dev.storj.io/trace/74ab99d10c3e98f0 FAIL (2025-12-25 12:26:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001784181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.575308909s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:25:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.575335611s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X...............................................X........] duration: 4.012870674s trace: https://jaeger.dev.storj.io/trace/c183c6332a8c273 FAIL (2025-12-25 12:25:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.012900289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.663093464s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:25:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.663125625s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)