Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.850746579s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:53:04 UTC) orbiter warning: context deadline exceeded Duration: 3.850771736s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X.....XX........X...X.X.................................] duration: 5.901021572s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:52:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.901048174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X.....XX........X...X.X................................] duration: 5.715801642s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:51:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.715829614s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.520430161s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:51:50 UTC) orbiter warning: context deadline exceeded Duration: 3.520459387s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.70060646s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:51:25 UTC) orbiter warning: context deadline exceeded Duration: 3.700707032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.....X.....XX........X...X.X...............................] duration: 5.668102187s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:51:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.668132526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.....X.....XX........X...X.X..............................] duration: 5.710075109s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:50:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.710100999s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.577232467s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:50:32 UTC) orbiter warning: context deadline exceeded Duration: 3.577261751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.....X.....XX........X...X.X.............................] duration: 5.66481252s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:49:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.66484451s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.508123139s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:49:26 UTC) orbiter warning: context deadline exceeded Duration: 3.508154138s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.493949512s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:48:50 UTC) orbiter warning: context deadline exceeded Duration: 3.493992462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.....X.....XX........X...X.X............................] duration: 5.655447877s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:48:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.655471522s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.....X.....XX........X...X.X...........................] duration: 5.681616888s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:48:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.681635693s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 4.314573665s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:47:44 UTC) orbiter warning: context deadline exceeded Duration: 4.314603742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X.....X.....XX........X...X.X..........................] duration: 5.662052032s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:47:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.6620822s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.596119187s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:47:24 UTC) orbiter warning: context deadline exceeded Duration: 3.596149354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X.....X.....XX........X...X.X.........................] duration: 5.741458888s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:46:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.741489577s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.519488341s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:46:57 UTC) orbiter warning: context deadline exceeded Duration: 3.519509741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X......X.....X.....XX........X...X.X........................] duration: 5.650430694s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:46:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.650458476s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X......X.....X.....XX........X...X.X.......................] duration: 5.771229506s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:46:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.771259562s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.612448412s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:45:53 UTC) orbiter warning: context deadline exceeded Duration: 3.612527533s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X......X.....X.....XX........X...X.X......................] duration: 5.707738057s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:45:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.707763927s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.641546731s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:45:25 UTC) orbiter warning: context deadline exceeded Duration: 3.641576659s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001711641s trace: https://jaeger.dev.storj.io/trace/6ea80a3d86918fdf FAIL (2025-12-25 11:45:10 UTC) orbiter warning: context deadline exceeded Duration: 5.001733795s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X..X......X.....X.....XX........X...X.X.....................] duration: 5.66209259s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:44:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.662162122s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XX..X......X.....X.....XX........X...X.X....................] duration: 5.668100323s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:43:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.668131891s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.XX..X......X.....X.....XX........X...X.X...................] duration: 5.720878558s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:42:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.72090585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..XX..X......X.....X.....XX........X...X.X..................] duration: 5.735244505s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:42:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.735276456s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...XX..X......X.....X.....XX........X...X.X.................] duration: 5.625503671s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:41:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.625530172s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....XX..X......X.....X.....XX........X...X.X................] duration: 5.723022027s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:40:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.723051272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....XX..X......X.....X.....XX........X...X.X...............] duration: 5.685101351s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:39:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.685127041s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......XX..X......X.....X.....XX........X...X.X..............] duration: 5.735098864s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:38:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.735124232s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......XX..X......X.....X.....XX........X...X.X.............] duration: 5.700795807s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:37:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.700816926s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........XX..X......X.....X.....XX........X...X.X............] duration: 5.717771522s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:37:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.717797742s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........XX..X......X.....X.....XX........X...X.X...........] duration: 5.898785754s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:35:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.898821641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........XX..X......X.....X.....XX........X...X.X..........] duration: 6.096041701s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:35:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 6.096110162s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........XX..X......X.....X.....XX........X...X.X.........] duration: 5.727536057s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:34:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.727558419s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [............XX..X......X.....X.....XX........X...X.X........] duration: 5.869309925s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:33:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.869323242s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.............XX..X......X.....X.....XX........X...X.X.......] duration: 6.676964961s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:33:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 6.676993735s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..............XX..X......X.....X.....XX........X...X.X......] duration: 5.739049942s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:32:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.739077514s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...............XX..X......X.....X.....XX........X...X.X.....] duration: 5.977902443s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:32:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.977931178s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [................XX..X......X.....X.....XX........X...X.X....] duration: 5.626696771s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:32:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.626723723s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.................XX..X......X.....X.....XX........X...X.X...] duration: 5.875978966s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:31:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.875992462s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................XX..X......X.....X.....XX........X...X.X..] duration: 5.870089362s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:30:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 5.870113389s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...................XX..X......X.....X.....XX........X...X.X.] duration: 7.33492085s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:30:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 7.334953142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X...................XX..X......X.....X.....XX........X...X.X] duration: 30.004011691s trace: https://jaeger.dev.storj.io/trace/50ede9c892a18d10 FAIL (2025-12-25 11:29:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A) context deadline exceeded ======LOGS====== 109416 (475h59m58.868192451s) [upload=417824]: Uploading segments... Dec 25 11:28:58.997347219 (475h59m58.868430254s) [upload=417824]: Got next segment. Inline: false Dec 25 11:28:58.997376434 (475h59m58.868459469s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:28:58.997380823 (475h59m58.868463858s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:28:59.243264977 (475h59m59.114348012s) [seg_pos={0 0}, upload=417824]: Begin upload segment... Dec 25 11:28:59.243318740 (475h59m59.114401775s) [seg_pos={0 0}, upload=417824]: Done begin upload segment. Dec 25 11:28:59.243378843 (475h59m59.114461878s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243407468 (475h59m59.114490513s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.243900928 (475h59m59.114983963s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244251305 (475h59m59.11533434s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244321578 (475h59m59.115404614s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244652619 (475h59m59.115735654s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.244964283 (475h59m59.116047318s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245610092 (475h59m59.116693137s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.245709892 (475h59m59.116792937s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246430605 (475h59m59.11751365s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.246982285 (475h59m59.118065321s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247330879 (475h59m59.118413914s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.247664915 (475h59m59.11874795s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.248091147 (475h59m59.119174192s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.249829638 (475h59m59.120912673s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:28:59.250050989 (475h59m59.121134024s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Uploading piece... Dec 25 11:29:00.965046439 (476h0m0.836129474s) [upload=417824]: Next returned nil segment Dec 25 11:29:00.965066868 (476h0m0.836149913s) [upload=417824]: Waiting for error group managing segments... Dec 25 11:29:01.254129941 (476h0m1.125212986s) [noise=true , address=145.79.227.15:4875 , piece_id=LZSEOFWHKKAL2B5RKJSK3JKE7O6GEGYOTTZDBCE4XCBDBPYFUUZA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254330073 (476h0m1.125413118s) [noise=true , address=145.79.227.24:4581 , piece_id=ZEYXPVKU6HDRM2LBGSDEQYVYI4WCYTWVFYWCR5ZKYVQZMDFE3YYQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.254792172 (476h0m1.125875207s) [noise=true , address=145.79.227.41:4085 , piece_id=KWCA3DHXAGA3LTP5N2Q5EDVXJAK2BCSZT4LLMQJVATEE62MIZIAA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258477239 (476h0m1.129560274s) [noise=true , address=145.79.227.32:4835 , piece_id=5R5CMAUMLAPH6FJXEYGED2KXXAX7Z65SH5NFETPZ2TGHL7KFOREA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.258987652 (476h0m1.130070697s) [noise=true , address=145.79.227.50:4375 , piece_id=ZGJYGFZUMSHQL7BP4HEDWIIDFFKUID6GU74R2U45IZP45DZNSVUA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.274237209 (476h0m1.145320254s) [noise=true , address=145.79.227.4:4186 , piece_id=GLOSGKJK2J3ZKUAJZY3SVLVQWK34J4M5ZUN6LVFHSSAAZY2DL62A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.277957052 (476h0m1.149040087s) [noise=true , address=145.79.227.48:4303 , piece_id=W6OEFEC35HNIUC6QHX2Z7VJNEK4CYLFPZHPODADZJQUOPJTI7GNQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.278705127 (476h0m1.149788172s) [noise=true , address=145.79.227.14:4837 , piece_id=R5TO2HEXBF3CICQUN4UT6K3HXOUWWYPF3MV77OJSWI4EQQHWSXFQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.285517168 (476h0m1.156600203s) [noise=true , address=145.79.227.29:4943 , piece_id=NVST7JN43HURTNCGA5KZ65R2IUSUGV3TVF664GZ2DQHP2XZYMRGQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.545902704 (476h0m1.416985749s) [noise=true , address=77.90.31.14:4663 , piece_id=BWZCLADLN6N2UEWD3QNK543TBGVZKE5ZVSCEGUSKRRN3AJTLBQFA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.608937203 (476h0m1.480020248s) [noise=true , address=45.91.116.11:4118 , piece_id=HLS54RV3463PX4D2YUKHFCDAWXNJLVTVXLDDZDULQKRXKCRM6BNA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.664837897 (476h0m1.535920942s) [noise=true , address=145.79.227.33:4869 , piece_id=SCEAGVGAR5OFEWICMQ6D2PKZYSB3JHMHGJPCXNOOXZT2US7OY5FQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918350406 (476h0m1.789433441s) [noise=true , address=45.91.116.8:4045 , piece_id=LK2P4LQZOY25WMXW4R4UTOGAMCYY7FZKSFR4WIDCSYUJHN2PKI4A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: <nil> Dec 25 11:29:01.918377498 (476h0m1.789460543s) [seg_pos={0 0}, upload=417824]: Segment reached optimal threshold of 13 pieces. Dec 25 11:29:01.918608657 (476h0m1.789691702s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59066->77.90.31.16:4736: use of closed network connection Dec 25 11:29:01.918618947 (476h0m1.789701993s) [noise=true , address=77.90.31.16:4736 , piece_id=YU327KPC5F2XENSOAOUE7XFVKEGA4MOCAX5DX3O2MYKVGRARXRRA, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918644014 (476h0m1.78972706s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40664->45.91.116.4:4937: use of closed network connection Dec 25 11:29:01.918656369 (476h0m1.789739414s) [noise=true , address=45.91.116.4:4937 , piece_id=7KYVUB7PGSRUA6MOPZUNYXWQJUFUOWDWT7BK4YSXLP42QR53UOXQ, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918917956 (476h0m1.790001001s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 11:29:01.918925711 (476h0m1.790008756s) [noise=true , address=45.91.116.10:4082 , piece_id=NYFOM6DP22XGXACPSRT76EWKIUCZSNW6ARKH7LCF6NOP7NC6JF7A, seg_pos={0 0}, upload=417824]: Long tail cancellation due to optimal threshold Dec 25 11:29:01.918943485 (476h0m1.79002653s) [seg_pos={0 0}, upload=417824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:29:01.919084654 (476h0m1.790167699s) [upload=417824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:29:01.919093781 (476h0m1.790176836s) [upload=417824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:29:02.226358593 (476h0m2.097441629s) [upload=417824]: Done uploading segments... Dec 25 11:29:02.226380095 (476h0m2.09746314s) [upload=417824]: Upload finished. err: <nil> Duration: 30.00411539s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XX...................XX..X......X.....X.....XX........X...X.] duration: 5.719635333s trace: https://jaeger.dev.storj.io/trace/266876f6696beca0 FAIL (2025-12-25 11:28:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: BH6PA3BQED6QFBOEGCBKPBDVLEPBOWIE74NK7Q5K265IRGVRUPEA) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: WX4ADCPVSR4GUPMNQ244U2S7MIW5MRITHTMSFDOECNXFEXUUHCTA) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: X5YDBLA3QFMWNNY5LZLOI72RYIM3KILYVEWUKELKOFAYFR57O56Q) context deadline exceeded ======LOGS====== Dec 25 11:27:31.549943619 (475h58m31.421026654s) [upload=417802]: Starting upload Dec 25 11:27:31.550041195 (475h58m31.42112423s) [upload=417802]: Done starting upload Dec 25 11:27:31.550097672 (475h58m31.421180717s) [upload=417802]: Uploading segments... Dec 25 11:27:31.550399878 (475h58m31.421482913s) [upload=417802]: Got next segment. Inline: false Dec 25 11:27:31.550451185 (475h58m31.42153423s) [upload=417802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:27:31.550457949 (475h58m31.421540994s) [upload=417802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:27:31.794661542 (475h58m31.665744577s) [seg_pos={0 0}, upload=417802]: Begin upload segment... Dec 25 11:27:31.794707600 (475h58m31.665790645s) [seg_pos={0 0}, upload=417802]: Done begin upload segment. Dec 25 11:27:31.794724543 (475h58m31.665807578s) [noise=true , address=145.79.227.50:4374 , piece_id=Q7ZEOSK3ANI3EC6M2RZMZMPI3LJF3O2E5NA7R67ISDEYBPEIYCMA, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.794735573 (475h58m31.665818608s) [noise=true , address=145.79.227.47:4266 , piece_id=XOOXIBGWLEJ4VCF3ZQW7ZI6EQLEGQXBHOVS55MWQUXMY3Z3BTVTQ, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.794797411 (475h58m31.665880456s) [noise=true , address=145.79.227.35:4481 , piece_id=NCVO7N7YG2LRTPSTX5E7MU3UYIY4TL4ZZJKT5PR2OGIK3E4HQOKA, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.795138650 (475h58m31.666221685s) [noise=true , address=145.79.227.51:4410 , piece_id=OF5QJKK2YBF7N5ORREU3KJ5NQX4QYSCHMIKTJBCMCEOR6NPSAQ4Q, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.795445284 (475h58m31.666528319s) [noise=true , address=45.91.116.7:4009 , piece_id=JS2XFKU63SKBFPF3SPJS5QHGIDR6BGFWDOVB72WD5ZCMW4XQ4HLA, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.795839936 (475h58m31.666922972s) [noise=true , address=145.79.227.37:4566 , piece_id=OCP4XTMMTHS25CZNOVDSWULI7ZUEWYUHSQFUUYYQMSIQ3V52Z6SQ, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.796145257 (475h58m31.667228292s) [noise=true , address=145.79.227.46:4602 , piece_id=2BEJAZQHCFQLOQTAVOCIVARLJG5SIO24XB5LUVHVDPQI5SUDHTBQ, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.796438005 (475h58m31.66752104s) [noise=true , address=145.79.227.32:4835 , piece_id=WX4ADCPVSR4GUPMNQ244U2S7MIW5MRITHTMSFDOECNXFEXUUHCTA, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.796763815 (475h58m31.66784685s) [noise=true , address=145.79.227.20:4437 , piece_id=CNI7SWU3WKTFECBYDI5NEBTYSN5B4IS2YHEVDX4YXHYIBVQLYEEQ, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.797097110 (475h58m31.668180155s) [noise=true , address=77.90.31.18:4808 , piece_id=WDVPFTKYCBRSAHTR3GPVO6LTHV7V65GD5DNIVVFPA7FEVY56W5JQ, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.797589868 (475h58m31.668672903s) [noise=true , address=77.90.31.19:4844 , piece_id=WW25ZXGPVW4HCQ7DZD65YZ2FJN2G7QO3KB7MLSATGN5EYMAOVTZQ, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.797983778 (475h58m31.669066813s) [noise=true , address=77.90.31.14:4664 , piece_id=X5YDBLA3QFMWNNY5LZLOI72RYIM3KILYVEWUKELKOFAYFR57O56Q, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.798321061 (475h58m31.669404096s) [noise=true , address=145.79.227.2:4114 , piece_id=TLNGBIIGN6SXJSQUIFS5X6K5GXQEOP77HU2PQ4NYEWLZYRBMCOCQ, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.798427543 (475h58m31.669510578s) [noise=true , address=145.79.227.43:4158 , piece_id=GF2IEDCBDE4CDIYU6J644IOBHXYZUHQSRMKR2QDCR3TCJCQTRGCA, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.798857512 (475h58m31.669940547s) [noise=true , address=45.91.116.6:4974 , piece_id=BH6PA3BQED6QFBOEGCBKPBDVLEPBOWIE74NK7Q5K265IRGVRUPEA, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.799185417 (475h58m31.670268452s) [noise=true , address=77.90.31.15:4700 , piece_id=CDDGHXO5HZ6FWAAC3SQUNVBI7DV5OYTP6ULG53PJ7K32EXLI3P3Q, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:33.508225870 (475h58m33.379308905s) [upload=417802]: Next returned nil segment Dec 25 11:27:33.508261127 (475h58m33.379344172s) [upload=417802]: Waiting for error group managing segments... Dec 25 11:27:33.788901351 (475h58m33.659984386s) [noise=true , address=145.79.227.37:4566 , piece_id=OCP4XTMMTHS25CZNOVDSWULI7ZUEWYUHSQFUUYYQMSIQ3V52Z6SQ, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.799457512 (475h58m33.670540547s) [noise=true , address=145.79.227.2:4114 , piece_id=TLNGBIIGN6SXJSQUIFS5X6K5GXQEOP77HU2PQ4NYEWLZYRBMCOCQ, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.799517396 (475h58m33.670600431s) [noise=true , address=145.79.227.35:4481 , piece_id=NCVO7N7YG2LRTPSTX5E7MU3UYIY4TL4ZZJKT5PR2OGIK3E4HQOKA, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.802833510 (475h58m33.673916545s) [noise=true , address=145.79.227.46:4602 , piece_id=2BEJAZQHCFQLOQTAVOCIVARLJG5SIO24XB5LUVHVDPQI5SUDHTBQ, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.802872635 (475h58m33.67395567s) [noise=true , address=145.79.227.20:4437 , piece_id=CNI7SWU3WKTFECBYDI5NEBTYSN5B4IS2YHEVDX4YXHYIBVQLYEEQ, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.820366677 (475h58m33.691449712s) [noise=true , address=145.79.227.50:4374 , piece_id=Q7ZEOSK3ANI3EC6M2RZMZMPI3LJF3O2E5NA7R67ISDEYBPEIYCMA, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.820926753 (475h58m33.692009798s) [noise=true , address=145.79.227.51:4410 , piece_id=OF5QJKK2YBF7N5ORREU3KJ5NQX4QYSCHMIKTJBCMCEOR6NPSAQ4Q, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.824979269 (475h58m33.696062315s) [noise=true , address=145.79.227.32:4835 , piece_id=WX4ADCPVSR4GUPMNQ244U2S7MIW5MRITHTMSFDOECNXFEXUUHCTA, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.829653972 (475h58m33.700737007s) [noise=true , address=145.79.227.43:4158 , piece_id=GF2IEDCBDE4CDIYU6J644IOBHXYZUHQSRMKR2QDCR3TCJCQTRGCA, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.830581908 (475h58m33.701664943s) [noise=true , address=145.79.227.47:4266 , piece_id=XOOXIBGWLEJ4VCF3ZQW7ZI6EQLEGQXBHOVS55MWQUXMY3Z3BTVTQ, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.990074067 (475h58m33.861157102s) [noise=true , address=77.90.31.14:4664 , piece_id=X5YDBLA3QFMWNNY5LZLOI72RYIM3KILYVEWUKELKOFAYFR57O56Q, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.992827149 (475h58m33.863910184s) [noise=true , address=77.90.31.15:4700 , piece_id=CDDGHXO5HZ6FWAAC3SQUNVBI7DV5OYTP6ULG53PJ7K32EXLI3P3Q, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:34.030383323 (475h58m33.901466358s) [noise=true , address=45.91.116.6:4974 , piece_id=BH6PA3BQED6QFBOEGCBKPBDVLEPBOWIE74NK7Q5K265IRGVRUPEA, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:34.030402328 (475h58m33.901485363s) [seg_pos={0 0}, upload=417802]: Segment reached optimal threshold of 13 pieces. Dec 25 11:27:34.030612619 (475h58m33.901695654s) [noise=true , address=45.91.116.7:4009 , piece_id=JS2XFKU63SKBFPF3SPJS5QHGIDR6BGFWDOVB72WD5ZCMW4XQ4HLA, seg_pos={0 0}, upload=417802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 25 11:27:34.030618901 (475h58m33.901701937s) [noise=true , address=45.91.116.7:4009 , piece_id=JS2XFKU63SKBFPF3SPJS5QHGIDR6BGFWDOVB72WD5ZCMW4XQ4HLA, seg_pos={0 0}, upload=417802]: Long tail cancellation due to optimal threshold Dec 25 11:27:34.030740883 (475h58m33.901823918s) [noise=true , address=77.90.31.18:4808 , piece_id=WDVPFTKYCBRSAHTR3GPVO6LTHV7V65GD5DNIVVFPA7FEVY56W5JQ, seg_pos={0 0}, upload=417802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 25 11:27:34.030745632 (475h58m33.901828667s) [noise=true , address=77.90.31.18:4808 , piece_id=WDVPFTKYCBRSAHTR3GPVO6LTHV7V65GD5DNIVVFPA7FEVY56W5JQ, seg_pos={0 0}, upload=417802]: Long tail cancellation due to optimal threshold Dec 25 11:27:34.030785478 (475h58m33.901868523s) [noise=true , address=77.90.31.19:4844 , piece_id=WW25ZXGPVW4HCQ7DZD65YZ2FJN2G7QO3KB7MLSATGN5EYMAOVTZQ, seg_pos={0 0}, upload=417802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 25 11:27:34.030803352 (475h58m33.901886387s) [noise=true , address=77.90.31.19:4844 , piece_id=WW25ZXGPVW4HCQ7DZD65YZ2FJN2G7QO3KB7MLSATGN5EYMAOVTZQ, seg_pos={0 0}, upload=417802]: Long tail cancellation due to optimal threshold Dec 25 11:27:34.030826617 (475h58m33.901909652s) [seg_pos={0 0}, upload=417802]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:27:34.030917409 (475h58m33.902000454s) [upload=417802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:27:34.030921928 (475h58m33.902004963s) [upload=417802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:27:34.344955910 (475h58m34.216038945s) [upload=417802]: Done uploading segments... Dec 25 11:27:34.344974384 (475h58m34.216057419s) [upload=417802]: Upload finished. err: <nil> Duration: 5.719686249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXX...................XX..X......X.....X.....XX........X...X] duration: 30.003449653s trace: https://jaeger.dev.storj.io/trace/266876f6696beca0 FAIL (2025-12-25 11:28:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: BH6PA3BQED6QFBOEGCBKPBDVLEPBOWIE74NK7Q5K265IRGVRUPEA) context deadline exceeded; piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: WX4ADCPVSR4GUPMNQ244U2S7MIW5MRITHTMSFDOECNXFEXUUHCTA) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: X5YDBLA3QFMWNNY5LZLOI72RYIM3KILYVEWUKELKOFAYFR57O56Q) context deadline exceeded ======LOGS====== Dec 25 11:27:31.549943619 (475h58m31.421026654s) [upload=417802]: Starting upload Dec 25 11:27:31.550041195 (475h58m31.42112423s) [upload=417802]: Done starting upload Dec 25 11:27:31.550097672 (475h58m31.421180717s) [upload=417802]: Uploading segments... Dec 25 11:27:31.550399878 (475h58m31.421482913s) [upload=417802]: Got next segment. Inline: false Dec 25 11:27:31.550451185 (475h58m31.42153423s) [upload=417802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:27:31.550457949 (475h58m31.421540994s) [upload=417802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:27:31.794661542 (475h58m31.665744577s) [seg_pos={0 0}, upload=417802]: Begin upload segment... Dec 25 11:27:31.794707600 (475h58m31.665790645s) [seg_pos={0 0}, upload=417802]: Done begin upload segment. Dec 25 11:27:31.794724543 (475h58m31.665807578s) [noise=true , address=145.79.227.50:4374 , piece_id=Q7ZEOSK3ANI3EC6M2RZMZMPI3LJF3O2E5NA7R67ISDEYBPEIYCMA, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.794735573 (475h58m31.665818608s) [noise=true , address=145.79.227.47:4266 , piece_id=XOOXIBGWLEJ4VCF3ZQW7ZI6EQLEGQXBHOVS55MWQUXMY3Z3BTVTQ, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.794797411 (475h58m31.665880456s) [noise=true , address=145.79.227.35:4481 , piece_id=NCVO7N7YG2LRTPSTX5E7MU3UYIY4TL4ZZJKT5PR2OGIK3E4HQOKA, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.795138650 (475h58m31.666221685s) [noise=true , address=145.79.227.51:4410 , piece_id=OF5QJKK2YBF7N5ORREU3KJ5NQX4QYSCHMIKTJBCMCEOR6NPSAQ4Q, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.795445284 (475h58m31.666528319s) [noise=true , address=45.91.116.7:4009 , piece_id=JS2XFKU63SKBFPF3SPJS5QHGIDR6BGFWDOVB72WD5ZCMW4XQ4HLA, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.795839936 (475h58m31.666922972s) [noise=true , address=145.79.227.37:4566 , piece_id=OCP4XTMMTHS25CZNOVDSWULI7ZUEWYUHSQFUUYYQMSIQ3V52Z6SQ, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.796145257 (475h58m31.667228292s) [noise=true , address=145.79.227.46:4602 , piece_id=2BEJAZQHCFQLOQTAVOCIVARLJG5SIO24XB5LUVHVDPQI5SUDHTBQ, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.796438005 (475h58m31.66752104s) [noise=true , address=145.79.227.32:4835 , piece_id=WX4ADCPVSR4GUPMNQ244U2S7MIW5MRITHTMSFDOECNXFEXUUHCTA, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.796763815 (475h58m31.66784685s) [noise=true , address=145.79.227.20:4437 , piece_id=CNI7SWU3WKTFECBYDI5NEBTYSN5B4IS2YHEVDX4YXHYIBVQLYEEQ, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.797097110 (475h58m31.668180155s) [noise=true , address=77.90.31.18:4808 , piece_id=WDVPFTKYCBRSAHTR3GPVO6LTHV7V65GD5DNIVVFPA7FEVY56W5JQ, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.797589868 (475h58m31.668672903s) [noise=true , address=77.90.31.19:4844 , piece_id=WW25ZXGPVW4HCQ7DZD65YZ2FJN2G7QO3KB7MLSATGN5EYMAOVTZQ, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.797983778 (475h58m31.669066813s) [noise=true , address=77.90.31.14:4664 , piece_id=X5YDBLA3QFMWNNY5LZLOI72RYIM3KILYVEWUKELKOFAYFR57O56Q, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.798321061 (475h58m31.669404096s) [noise=true , address=145.79.227.2:4114 , piece_id=TLNGBIIGN6SXJSQUIFS5X6K5GXQEOP77HU2PQ4NYEWLZYRBMCOCQ, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.798427543 (475h58m31.669510578s) [noise=true , address=145.79.227.43:4158 , piece_id=GF2IEDCBDE4CDIYU6J644IOBHXYZUHQSRMKR2QDCR3TCJCQTRGCA, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.798857512 (475h58m31.669940547s) [noise=true , address=45.91.116.6:4974 , piece_id=BH6PA3BQED6QFBOEGCBKPBDVLEPBOWIE74NK7Q5K265IRGVRUPEA, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:31.799185417 (475h58m31.670268452s) [noise=true , address=77.90.31.15:4700 , piece_id=CDDGHXO5HZ6FWAAC3SQUNVBI7DV5OYTP6ULG53PJ7K32EXLI3P3Q, seg_pos={0 0}, upload=417802]: Uploading piece... Dec 25 11:27:33.508225870 (475h58m33.379308905s) [upload=417802]: Next returned nil segment Dec 25 11:27:33.508261127 (475h58m33.379344172s) [upload=417802]: Waiting for error group managing segments... Dec 25 11:27:33.788901351 (475h58m33.659984386s) [noise=true , address=145.79.227.37:4566 , piece_id=OCP4XTMMTHS25CZNOVDSWULI7ZUEWYUHSQFUUYYQMSIQ3V52Z6SQ, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.799457512 (475h58m33.670540547s) [noise=true , address=145.79.227.2:4114 , piece_id=TLNGBIIGN6SXJSQUIFS5X6K5GXQEOP77HU2PQ4NYEWLZYRBMCOCQ, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.799517396 (475h58m33.670600431s) [noise=true , address=145.79.227.35:4481 , piece_id=NCVO7N7YG2LRTPSTX5E7MU3UYIY4TL4ZZJKT5PR2OGIK3E4HQOKA, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.802833510 (475h58m33.673916545s) [noise=true , address=145.79.227.46:4602 , piece_id=2BEJAZQHCFQLOQTAVOCIVARLJG5SIO24XB5LUVHVDPQI5SUDHTBQ, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.802872635 (475h58m33.67395567s) [noise=true , address=145.79.227.20:4437 , piece_id=CNI7SWU3WKTFECBYDI5NEBTYSN5B4IS2YHEVDX4YXHYIBVQLYEEQ, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.820366677 (475h58m33.691449712s) [noise=true , address=145.79.227.50:4374 , piece_id=Q7ZEOSK3ANI3EC6M2RZMZMPI3LJF3O2E5NA7R67ISDEYBPEIYCMA, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.820926753 (475h58m33.692009798s) [noise=true , address=145.79.227.51:4410 , piece_id=OF5QJKK2YBF7N5ORREU3KJ5NQX4QYSCHMIKTJBCMCEOR6NPSAQ4Q, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.824979269 (475h58m33.696062315s) [noise=true , address=145.79.227.32:4835 , piece_id=WX4ADCPVSR4GUPMNQ244U2S7MIW5MRITHTMSFDOECNXFEXUUHCTA, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.829653972 (475h58m33.700737007s) [noise=true , address=145.79.227.43:4158 , piece_id=GF2IEDCBDE4CDIYU6J644IOBHXYZUHQSRMKR2QDCR3TCJCQTRGCA, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.830581908 (475h58m33.701664943s) [noise=true , address=145.79.227.47:4266 , piece_id=XOOXIBGWLEJ4VCF3ZQW7ZI6EQLEGQXBHOVS55MWQUXMY3Z3BTVTQ, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.990074067 (475h58m33.861157102s) [noise=true , address=77.90.31.14:4664 , piece_id=X5YDBLA3QFMWNNY5LZLOI72RYIM3KILYVEWUKELKOFAYFR57O56Q, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:33.992827149 (475h58m33.863910184s) [noise=true , address=77.90.31.15:4700 , piece_id=CDDGHXO5HZ6FWAAC3SQUNVBI7DV5OYTP6ULG53PJ7K32EXLI3P3Q, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:34.030383323 (475h58m33.901466358s) [noise=true , address=45.91.116.6:4974 , piece_id=BH6PA3BQED6QFBOEGCBKPBDVLEPBOWIE74NK7Q5K265IRGVRUPEA, seg_pos={0 0}, upload=417802]: Done uploading piece. err: <nil> Dec 25 11:27:34.030402328 (475h58m33.901485363s) [seg_pos={0 0}, upload=417802]: Segment reached optimal threshold of 13 pieces. Dec 25 11:27:34.030612619 (475h58m33.901695654s) [noise=true , address=45.91.116.7:4009 , piece_id=JS2XFKU63SKBFPF3SPJS5QHGIDR6BGFWDOVB72WD5ZCMW4XQ4HLA, seg_pos={0 0}, upload=417802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 25 11:27:34.030618901 (475h58m33.901701937s) [noise=true , address=45.91.116.7:4009 , piece_id=JS2XFKU63SKBFPF3SPJS5QHGIDR6BGFWDOVB72WD5ZCMW4XQ4HLA, seg_pos={0 0}, upload=417802]: Long tail cancellation due to optimal threshold Dec 25 11:27:34.030740883 (475h58m33.901823918s) [noise=true , address=77.90.31.18:4808 , piece_id=WDVPFTKYCBRSAHTR3GPVO6LTHV7V65GD5DNIVVFPA7FEVY56W5JQ, seg_pos={0 0}, upload=417802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 25 11:27:34.030745632 (475h58m33.901828667s) [noise=true , address=77.90.31.18:4808 , piece_id=WDVPFTKYCBRSAHTR3GPVO6LTHV7V65GD5DNIVVFPA7FEVY56W5JQ, seg_pos={0 0}, upload=417802]: Long tail cancellation due to optimal threshold Dec 25 11:27:34.030785478 (475h58m33.901868523s) [noise=true , address=77.90.31.19:4844 , piece_id=WW25ZXGPVW4HCQ7DZD65YZ2FJN2G7QO3KB7MLSATGN5EYMAOVTZQ, seg_pos={0 0}, upload=417802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 25 11:27:34.030803352 (475h58m33.901886387s) [noise=true , address=77.90.31.19:4844 , piece_id=WW25ZXGPVW4HCQ7DZD65YZ2FJN2G7QO3KB7MLSATGN5EYMAOVTZQ, seg_pos={0 0}, upload=417802]: Long tail cancellation due to optimal threshold Dec 25 11:27:34.030826617 (475h58m33.901909652s) [seg_pos={0 0}, upload=417802]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:27:34.030917409 (475h58m33.902000454s) [upload=417802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:27:34.030921928 (475h58m33.902004963s) [upload=417802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:27:34.344955910 (475h58m34.216038945s) [upload=417802]: Done uploading segments... Dec 25 11:27:34.344974384 (475h58m34.216057419s) [upload=417802]: Upload finished. err: <nil> Duration: 30.003470984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.XXX...................XX..X......X.....X.....XX........X...] duration: 6.180876491s trace: https://jaeger.dev.storj.io/trace/1c3adc0244de0406 FAIL (2025-12-25 11:26:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 11:23:46.314018079 (475h54m46.185101114s) [upload=417745]: Starting upload Dec 25 11:23:46.314111898 (475h54m46.185194943s) [upload=417745]: Done starting upload Dec 25 11:23:46.314403052 (475h54m46.185486097s) [upload=417745]: Uploading segments... Dec 25 11:23:46.314445523 (475h54m46.185528568s) [upload=417745]: Got next segment. Inline: false Dec 25 11:23:46.314474157 (475h54m46.185557192s) [upload=417745]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:23:46.314478276 (475h54m46.185561311s) [upload=417745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:23:46.558402187 (475h54m46.429485222s) [seg_pos={0 0}, upload=417745]: Begin upload segment... Dec 25 11:23:46.558491406 (475h54m46.429574451s) [seg_pos={0 0}, upload=417745]: Done begin upload segment. Dec 25 11:23:46.558564906 (475h54m46.429647971s) [noise=true , address=145.79.227.14:4839 , piece_id=QHZZFVADSC7CPMGHQSCYSF664PH3SYL2WLFQDOD2QOVUEUXM3XTA, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.558625762 (475h54m46.429708797s) [noise=true , address=145.79.227.51:4411 , piece_id=BH6PJJYRYHTVKKETDDHNUYMYOC3MGEMXZOX6S4FXIKBFGLVTTJFA, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.559216878 (475h54m46.430299923s) [noise=true , address=145.79.227.2:4114 , piece_id=JLADII3UTANHMPN7JRID4VNATO3JFJLACYHJKHHFGW72LBSDBV2Q, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.559618712 (475h54m46.430701747s) [noise=true , address=145.79.227.40:4050 , piece_id=Y3ZYOOUQMX376XFRICKHRQDZTL3FC4RYNCDBJTTMIODSV5UU2WWQ, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.560027060 (475h54m46.431110105s) [noise=true , address=45.91.116.6:4973 , piece_id=47HSFB4B7FLPTYMKO7EHBGZKTHFZAQVVSMCRMG32KA7YGWSAFV7Q, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.560448993 (475h54m46.431532039s) [noise=true , address=145.79.227.28:4727 , piece_id=QJTT2GDFMDBJXYAOSQNXGO5QWVDZKELYGHIUIXLCDPOODDOYTGRA, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.560856460 (475h54m46.431939505s) [noise=true , address=145.79.227.33:4869 , piece_id=GWMPNUGU7UM7KPTBJGGBFBQQQOE3PUG6BC62TH2QZXR3UHKPMLJA, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.561397760 (475h54m46.432480805s) [noise=true , address=145.79.227.11:4732 , piece_id=CLAHZM5YV7KTAU66T5JOSMFUOFM53HXI3ILXBIMPTOZ7LOEUQJOQ, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.561915295 (475h54m46.43299833s) [noise=true , address=145.79.227.47:4266 , piece_id=XQYUXZ7U5VFCL44YEJEQ5ZECZRU7E7MCY2MAO2YYUHO5PNF57FHQ, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.563256159 (475h54m46.434339204s) [noise=true , address=145.79.227.46:4603 , piece_id=GQCY3XQBWQNUFTTG3ZZVO5P5CJYXSRWKP52IWDQG7MDAFHHDXNQQ, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.564651205 (475h54m46.435734251s) [noise=true , address=45.91.116.8:4048 , piece_id=VUVR3D3VUFB3PT6JNYHPALSFG2LGNN7AMEFYDBEEP5FKEOGBD2GA, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.567296773 (475h54m46.438379818s) [noise=true , address=45.91.116.12:4225 , piece_id=WLUFTPEXYVKJEM5YWBUJD23KLWJBCFSSDLFIND7SNCGKPRQ2KTJA, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.569380992 (475h54m46.440464037s) [noise=true , address=145.79.227.19:4077 , piece_id=PDQQRHMBXIHN6JUJNR5YHXGT7554Q37ZEEFNYHN4VRDTSCB75QOA, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.571059558 (475h54m46.442142593s) [noise=true , address=45.91.116.9:4263 , piece_id=A7GX4VLOTAXTVI3E4RSRXI3HQXCIV7AG42XZCOLKJBULRLEQ2CEQ, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.572433955 (475h54m46.443517s) [noise=true , address=145.79.227.49:4337 , piece_id=CWAEYO4UHZFFGV2ULDJSTTAGZU5G7SOOE2GRP66IRZVJMX4TB2BQ, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.573039598 (475h54m46.444122633s) [noise=true , address=145.79.227.31:4799 , piece_id=N4XGVXR7JERCBSLRIITWPVEYAYWBDMWUNRD2SLFX62ZLGAAEGOWQ, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:48.277450726 (475h54m48.148533771s) [upload=417745]: Next returned nil segment Dec 25 11:23:48.277466456 (475h54m48.148549491s) [upload=417745]: Waiting for error group managing segments... Dec 25 11:23:48.554028045 (475h54m48.42511109s) [noise=true , address=145.79.227.33:4869 , piece_id=GWMPNUGU7UM7KPTBJGGBFBQQQOE3PUG6BC62TH2QZXR3UHKPMLJA, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.560406801 (475h54m48.431489846s) [noise=true , address=145.79.227.51:4411 , piece_id=BH6PJJYRYHTVKKETDDHNUYMYOC3MGEMXZOX6S4FXIKBFGLVTTJFA, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.565406201 (475h54m48.436489246s) [noise=true , address=145.79.227.47:4266 , piece_id=XQYUXZ7U5VFCL44YEJEQ5ZECZRU7E7MCY2MAO2YYUHO5PNF57FHQ, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.573910615 (475h54m48.44499365s) [noise=true , address=145.79.227.11:4732 , piece_id=CLAHZM5YV7KTAU66T5JOSMFUOFM53HXI3ILXBIMPTOZ7LOEUQJOQ, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.577531049 (475h54m48.448614094s) [noise=true , address=145.79.227.19:4077 , piece_id=PDQQRHMBXIHN6JUJNR5YHXGT7554Q37ZEEFNYHN4VRDTSCB75QOA, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.578231382 (475h54m48.449314427s) [noise=true , address=145.79.227.46:4603 , piece_id=GQCY3XQBWQNUFTTG3ZZVO5P5CJYXSRWKP52IWDQG7MDAFHHDXNQQ, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.587986407 (475h54m48.459069442s) [noise=true , address=145.79.227.28:4727 , piece_id=QJTT2GDFMDBJXYAOSQNXGO5QWVDZKELYGHIUIXLCDPOODDOYTGRA, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.588157733 (475h54m48.459240779s) [noise=true , address=145.79.227.2:4114 , piece_id=JLADII3UTANHMPN7JRID4VNATO3JFJLACYHJKHHFGW72LBSDBV2Q, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.603017339 (475h54m48.474100374s) [noise=true , address=145.79.227.14:4839 , piece_id=QHZZFVADSC7CPMGHQSCYSF664PH3SYL2WLFQDOD2QOVUEUXM3XTA, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.604830010 (475h54m48.475913055s) [noise=true , address=145.79.227.31:4799 , piece_id=N4XGVXR7JERCBSLRIITWPVEYAYWBDMWUNRD2SLFX62ZLGAAEGOWQ, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.608997856 (475h54m48.480080901s) [noise=true , address=145.79.227.49:4337 , piece_id=CWAEYO4UHZFFGV2ULDJSTTAGZU5G7SOOE2GRP66IRZVJMX4TB2BQ, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.852986240 (475h54m48.724069275s) [noise=true , address=45.91.116.9:4263 , piece_id=A7GX4VLOTAXTVI3E4RSRXI3HQXCIV7AG42XZCOLKJBULRLEQ2CEQ, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.938824260 (475h54m48.809907295s) [noise=true , address=45.91.116.8:4048 , piece_id=VUVR3D3VUFB3PT6JNYHPALSFG2LGNN7AMEFYDBEEP5FKEOGBD2GA, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.938843917 (475h54m48.809926952s) [seg_pos={0 0}, upload=417745]: Segment reached optimal threshold of 13 pieces. Dec 25 11:23:48.939100766 (475h54m48.810183801s) [noise=true , address=45.91.116.6:4973 , piece_id=47HSFB4B7FLPTYMKO7EHBGZKTHFZAQVVSMCRMG32KA7YGWSAFV7Q, seg_pos={0 0}, upload=417745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: read tcp 188.166.225.170:40864->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 11:23:48.939107639 (475h54m48.810190684s) [noise=true , address=45.91.116.6:4973 , piece_id=47HSFB4B7FLPTYMKO7EHBGZKTHFZAQVVSMCRMG32KA7YGWSAFV7Q, seg_pos={0 0}, upload=417745]: Long tail cancellation due to optimal threshold Dec 25 11:23:48.939179316 (475h54m48.810262351s) [noise=true , address=145.79.227.40:4050 , piece_id=Y3ZYOOUQMX376XFRICKHRQDZTL3FC4RYNCDBJTTMIODSV5UU2WWQ, seg_pos={0 0}, upload=417745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 25 11:23:48.939194284 (475h54m48.81027732s) [noise=true , address=145.79.227.40:4050 , piece_id=Y3ZYOOUQMX376XFRICKHRQDZTL3FC4RYNCDBJTTMIODSV5UU2WWQ, seg_pos={0 0}, upload=417745]: Long tail cancellation due to optimal threshold Dec 25 11:23:48.939432508 (475h54m48.810515553s) [noise=true , address=45.91.116.12:4225 , piece_id=WLUFTPEXYVKJEM5YWBUJD23KLWJBCFSSDLFIND7SNCGKPRQ2KTJA, seg_pos={0 0}, upload=417745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: context canceled; context canceled Dec 25 11:23:48.939441836 (475h54m48.810524871s) [noise=true , address=45.91.116.12:4225 , piece_id=WLUFTPEXYVKJEM5YWBUJD23KLWJBCFSSDLFIND7SNCGKPRQ2KTJA, seg_pos={0 0}, upload=417745]: Long tail cancellation due to optimal threshold Dec 25 11:23:48.939456253 (475h54m48.810539288s) [seg_pos={0 0}, upload=417745]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:23:48.939522259 (475h54m48.810605304s) [upload=417745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:23:48.939526797 (475h54m48.810609832s) [upload=417745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:23:49.235671566 (475h54m49.106754601s) [upload=417745]: Done uploading segments... Dec 25 11:23:49.235691614 (475h54m49.106774659s) [upload=417745]: Upload finished. err: <nil> Duration: 6.180905526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..XXX...................XX..X......X.....X.....XX........X..] duration: 5.720238651s trace: https://jaeger.dev.storj.io/trace/1c3adc0244de0406 FAIL (2025-12-25 11:25:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 11:23:46.314018079 (475h54m46.185101114s) [upload=417745]: Starting upload Dec 25 11:23:46.314111898 (475h54m46.185194943s) [upload=417745]: Done starting upload Dec 25 11:23:46.314403052 (475h54m46.185486097s) [upload=417745]: Uploading segments... Dec 25 11:23:46.314445523 (475h54m46.185528568s) [upload=417745]: Got next segment. Inline: false Dec 25 11:23:46.314474157 (475h54m46.185557192s) [upload=417745]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:23:46.314478276 (475h54m46.185561311s) [upload=417745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:23:46.558402187 (475h54m46.429485222s) [seg_pos={0 0}, upload=417745]: Begin upload segment... Dec 25 11:23:46.558491406 (475h54m46.429574451s) [seg_pos={0 0}, upload=417745]: Done begin upload segment. Dec 25 11:23:46.558564906 (475h54m46.429647971s) [noise=true , address=145.79.227.14:4839 , piece_id=QHZZFVADSC7CPMGHQSCYSF664PH3SYL2WLFQDOD2QOVUEUXM3XTA, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.558625762 (475h54m46.429708797s) [noise=true , address=145.79.227.51:4411 , piece_id=BH6PJJYRYHTVKKETDDHNUYMYOC3MGEMXZOX6S4FXIKBFGLVTTJFA, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.559216878 (475h54m46.430299923s) [noise=true , address=145.79.227.2:4114 , piece_id=JLADII3UTANHMPN7JRID4VNATO3JFJLACYHJKHHFGW72LBSDBV2Q, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.559618712 (475h54m46.430701747s) [noise=true , address=145.79.227.40:4050 , piece_id=Y3ZYOOUQMX376XFRICKHRQDZTL3FC4RYNCDBJTTMIODSV5UU2WWQ, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.560027060 (475h54m46.431110105s) [noise=true , address=45.91.116.6:4973 , piece_id=47HSFB4B7FLPTYMKO7EHBGZKTHFZAQVVSMCRMG32KA7YGWSAFV7Q, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.560448993 (475h54m46.431532039s) [noise=true , address=145.79.227.28:4727 , piece_id=QJTT2GDFMDBJXYAOSQNXGO5QWVDZKELYGHIUIXLCDPOODDOYTGRA, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.560856460 (475h54m46.431939505s) [noise=true , address=145.79.227.33:4869 , piece_id=GWMPNUGU7UM7KPTBJGGBFBQQQOE3PUG6BC62TH2QZXR3UHKPMLJA, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.561397760 (475h54m46.432480805s) [noise=true , address=145.79.227.11:4732 , piece_id=CLAHZM5YV7KTAU66T5JOSMFUOFM53HXI3ILXBIMPTOZ7LOEUQJOQ, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.561915295 (475h54m46.43299833s) [noise=true , address=145.79.227.47:4266 , piece_id=XQYUXZ7U5VFCL44YEJEQ5ZECZRU7E7MCY2MAO2YYUHO5PNF57FHQ, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.563256159 (475h54m46.434339204s) [noise=true , address=145.79.227.46:4603 , piece_id=GQCY3XQBWQNUFTTG3ZZVO5P5CJYXSRWKP52IWDQG7MDAFHHDXNQQ, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.564651205 (475h54m46.435734251s) [noise=true , address=45.91.116.8:4048 , piece_id=VUVR3D3VUFB3PT6JNYHPALSFG2LGNN7AMEFYDBEEP5FKEOGBD2GA, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.567296773 (475h54m46.438379818s) [noise=true , address=45.91.116.12:4225 , piece_id=WLUFTPEXYVKJEM5YWBUJD23KLWJBCFSSDLFIND7SNCGKPRQ2KTJA, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.569380992 (475h54m46.440464037s) [noise=true , address=145.79.227.19:4077 , piece_id=PDQQRHMBXIHN6JUJNR5YHXGT7554Q37ZEEFNYHN4VRDTSCB75QOA, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.571059558 (475h54m46.442142593s) [noise=true , address=45.91.116.9:4263 , piece_id=A7GX4VLOTAXTVI3E4RSRXI3HQXCIV7AG42XZCOLKJBULRLEQ2CEQ, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.572433955 (475h54m46.443517s) [noise=true , address=145.79.227.49:4337 , piece_id=CWAEYO4UHZFFGV2ULDJSTTAGZU5G7SOOE2GRP66IRZVJMX4TB2BQ, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:46.573039598 (475h54m46.444122633s) [noise=true , address=145.79.227.31:4799 , piece_id=N4XGVXR7JERCBSLRIITWPVEYAYWBDMWUNRD2SLFX62ZLGAAEGOWQ, seg_pos={0 0}, upload=417745]: Uploading piece... Dec 25 11:23:48.277450726 (475h54m48.148533771s) [upload=417745]: Next returned nil segment Dec 25 11:23:48.277466456 (475h54m48.148549491s) [upload=417745]: Waiting for error group managing segments... Dec 25 11:23:48.554028045 (475h54m48.42511109s) [noise=true , address=145.79.227.33:4869 , piece_id=GWMPNUGU7UM7KPTBJGGBFBQQQOE3PUG6BC62TH2QZXR3UHKPMLJA, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.560406801 (475h54m48.431489846s) [noise=true , address=145.79.227.51:4411 , piece_id=BH6PJJYRYHTVKKETDDHNUYMYOC3MGEMXZOX6S4FXIKBFGLVTTJFA, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.565406201 (475h54m48.436489246s) [noise=true , address=145.79.227.47:4266 , piece_id=XQYUXZ7U5VFCL44YEJEQ5ZECZRU7E7MCY2MAO2YYUHO5PNF57FHQ, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.573910615 (475h54m48.44499365s) [noise=true , address=145.79.227.11:4732 , piece_id=CLAHZM5YV7KTAU66T5JOSMFUOFM53HXI3ILXBIMPTOZ7LOEUQJOQ, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.577531049 (475h54m48.448614094s) [noise=true , address=145.79.227.19:4077 , piece_id=PDQQRHMBXIHN6JUJNR5YHXGT7554Q37ZEEFNYHN4VRDTSCB75QOA, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.578231382 (475h54m48.449314427s) [noise=true , address=145.79.227.46:4603 , piece_id=GQCY3XQBWQNUFTTG3ZZVO5P5CJYXSRWKP52IWDQG7MDAFHHDXNQQ, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.587986407 (475h54m48.459069442s) [noise=true , address=145.79.227.28:4727 , piece_id=QJTT2GDFMDBJXYAOSQNXGO5QWVDZKELYGHIUIXLCDPOODDOYTGRA, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.588157733 (475h54m48.459240779s) [noise=true , address=145.79.227.2:4114 , piece_id=JLADII3UTANHMPN7JRID4VNATO3JFJLACYHJKHHFGW72LBSDBV2Q, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.603017339 (475h54m48.474100374s) [noise=true , address=145.79.227.14:4839 , piece_id=QHZZFVADSC7CPMGHQSCYSF664PH3SYL2WLFQDOD2QOVUEUXM3XTA, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.604830010 (475h54m48.475913055s) [noise=true , address=145.79.227.31:4799 , piece_id=N4XGVXR7JERCBSLRIITWPVEYAYWBDMWUNRD2SLFX62ZLGAAEGOWQ, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.608997856 (475h54m48.480080901s) [noise=true , address=145.79.227.49:4337 , piece_id=CWAEYO4UHZFFGV2ULDJSTTAGZU5G7SOOE2GRP66IRZVJMX4TB2BQ, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.852986240 (475h54m48.724069275s) [noise=true , address=45.91.116.9:4263 , piece_id=A7GX4VLOTAXTVI3E4RSRXI3HQXCIV7AG42XZCOLKJBULRLEQ2CEQ, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.938824260 (475h54m48.809907295s) [noise=true , address=45.91.116.8:4048 , piece_id=VUVR3D3VUFB3PT6JNYHPALSFG2LGNN7AMEFYDBEEP5FKEOGBD2GA, seg_pos={0 0}, upload=417745]: Done uploading piece. err: <nil> Dec 25 11:23:48.938843917 (475h54m48.809926952s) [seg_pos={0 0}, upload=417745]: Segment reached optimal threshold of 13 pieces. Dec 25 11:23:48.939100766 (475h54m48.810183801s) [noise=true , address=45.91.116.6:4973 , piece_id=47HSFB4B7FLPTYMKO7EHBGZKTHFZAQVVSMCRMG32KA7YGWSAFV7Q, seg_pos={0 0}, upload=417745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: read tcp 188.166.225.170:40864->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 11:23:48.939107639 (475h54m48.810190684s) [noise=true , address=45.91.116.6:4973 , piece_id=47HSFB4B7FLPTYMKO7EHBGZKTHFZAQVVSMCRMG32KA7YGWSAFV7Q, seg_pos={0 0}, upload=417745]: Long tail cancellation due to optimal threshold Dec 25 11:23:48.939179316 (475h54m48.810262351s) [noise=true , address=145.79.227.40:4050 , piece_id=Y3ZYOOUQMX376XFRICKHRQDZTL3FC4RYNCDBJTTMIODSV5UU2WWQ, seg_pos={0 0}, upload=417745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 25 11:23:48.939194284 (475h54m48.81027732s) [noise=true , address=145.79.227.40:4050 , piece_id=Y3ZYOOUQMX376XFRICKHRQDZTL3FC4RYNCDBJTTMIODSV5UU2WWQ, seg_pos={0 0}, upload=417745]: Long tail cancellation due to optimal threshold Dec 25 11:23:48.939432508 (475h54m48.810515553s) [noise=true , address=45.91.116.12:4225 , piece_id=WLUFTPEXYVKJEM5YWBUJD23KLWJBCFSSDLFIND7SNCGKPRQ2KTJA, seg_pos={0 0}, upload=417745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: context canceled; context canceled Dec 25 11:23:48.939441836 (475h54m48.810524871s) [noise=true , address=45.91.116.12:4225 , piece_id=WLUFTPEXYVKJEM5YWBUJD23KLWJBCFSSDLFIND7SNCGKPRQ2KTJA, seg_pos={0 0}, upload=417745]: Long tail cancellation due to optimal threshold Dec 25 11:23:48.939456253 (475h54m48.810539288s) [seg_pos={0 0}, upload=417745]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:23:48.939522259 (475h54m48.810605304s) [upload=417745]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:23:48.939526797 (475h54m48.810609832s) [upload=417745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:23:49.235671566 (475h54m49.106754601s) [upload=417745]: Done uploading segments... Dec 25 11:23:49.235691614 (475h54m49.106774659s) [upload=417745]: Upload finished. err: <nil> Duration: 5.72026911s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)