Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 4.527829259s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:04:23 UTC) orbiter warning: context deadline exceeded Duration: 4.527856109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X...X.X...............................................] duration: 5.669509374s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 12:03:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.669536976s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.439210121s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:03:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 6.439236051s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.549153165s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:03:37 UTC) orbiter warning: context deadline exceeded Duration: 3.549184884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X...X.X..............................................] duration: 5.688248685s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 12:03:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.688274424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X...X.X.............................................] duration: 5.715641348s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 12:03:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.715672117s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.709958657s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:02:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.709984497s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.883382213s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:02:32 UTC) orbiter warning: context deadline exceeded Duration: 3.883409837s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.55555635s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:02:16 UTC) orbiter warning: context deadline exceeded Duration: 3.555581827s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.501827502s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:02:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.501845916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X........X...X.X............................................] duration: 5.781578204s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 12:02:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.78160798s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 4.563387967s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:02:00 UTC) orbiter warning: context deadline exceeded Duration: 4.563417714s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.632823678s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:01:28 UTC) orbiter warning: context deadline exceeded Duration: 3.632846421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XX........X...X.X...........................................] duration: 6.001540666s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 12:01:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 6.001615899s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.082340886s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:00:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 6.082372506s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 5.001203432s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 12:00:16 UTC) orbiter warning: context deadline exceeded Duration: 5.001238627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XX........X...X.X..........................................] duration: 5.802517395s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:59:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.802595284s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.524395874s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:59:50 UTC) orbiter warning: context deadline exceeded Duration: 3.524423226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.556763159s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 11:59:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.556775071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 47.92722723s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 11:59:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 47.927240796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.455539178s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:59:08 UTC) orbiter warning: context deadline exceeded Duration: 3.455565509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX........X...X.X.........................................] duration: 5.72002185s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:58:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.720045495s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 56.541442014s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 11:58:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 56.541457473s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.360823122s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:58:47 UTC) orbiter warning: context deadline exceeded Duration: 3.360847729s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XX........X...X.X........................................] duration: 5.953117824s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:58:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.95319888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 56.516506939s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 11:57:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 56.516523301s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.564323574s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:57:51 UTC) orbiter warning: context deadline exceeded Duration: 3.564349502s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XX........X...X.X.......................................] duration: 5.708591576s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:56:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.708617286s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.00276999s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 11:56:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 1m0.00278012s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 4.481832632s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:56:37 UTC) orbiter warning: context deadline exceeded Duration: 4.481852452s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....XX........X...X.X......................................] duration: 5.689367547s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:56:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.689395693s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.420768892s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:55:32 UTC) orbiter warning: context deadline exceeded Duration: 3.420791446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.....XX........X...X.X.....................................] duration: 5.69816479s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:55:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.698187232s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.....XX........X...X.X....................................] duration: 5.727512944s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:55:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.727539314s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.442598418s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:55:01 UTC) orbiter warning: context deadline exceeded Duration: 3.443510767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.....XX........X...X.X...................................] duration: 5.710573208s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:54:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.71060087s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.521266507s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:53:49 UTC) orbiter warning: context deadline exceeded Duration: 3.521291305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.....XX........X...X.X..................................] duration: 5.658496026s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:53:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.658527315s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.850746579s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:53:04 UTC) orbiter warning: context deadline exceeded Duration: 3.850771736s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X.....XX........X...X.X.................................] duration: 5.901021572s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:52:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.901048174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X.....XX........X...X.X................................] duration: 5.715801642s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:51:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.715829614s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.520430161s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:51:50 UTC) orbiter warning: context deadline exceeded Duration: 3.520459387s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.70060646s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:51:25 UTC) orbiter warning: context deadline exceeded Duration: 3.700707032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.....X.....XX........X...X.X...............................] duration: 5.668102187s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:51:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.668132526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.....X.....XX........X...X.X..............................] duration: 5.710075109s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:50:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.710100999s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.577232467s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:50:32 UTC) orbiter warning: context deadline exceeded Duration: 3.577261751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.....X.....XX........X...X.X.............................] duration: 5.66481252s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:49:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.66484451s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.508123139s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:49:26 UTC) orbiter warning: context deadline exceeded Duration: 3.508154138s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.493949512s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:48:50 UTC) orbiter warning: context deadline exceeded Duration: 3.493992462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.....X.....XX........X...X.X............................] duration: 5.655447877s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:48:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.655471522s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)