Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X....X.......X.......X..............................X.X.X.] duration: 24.489033418s trace: https://jaeger.dev.storj.io/trace/136f097503400aab FAIL (2025-12-26 23:20:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: HBNDBNN5ESS3RPZBTZIWWPM4WSQC6HDAZLURIBRJTNCXBDZ5DYVQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: BIDIVS73X6IL4GAVPMG4IDGFBH5MICS2GXVBR422V6NLHGT7QCNA) context deadline exceeded ======LOGS====== 23:19:12.466635016 (511h50m12.337718061s) [upload=449126]: Starting upload Dec 26 23:19:12.466708205 (511h50m12.337791251s) [upload=449126]: Done starting upload Dec 26 23:19:12.466748342 (511h50m12.337831377s) [upload=449126]: Uploading segments... Dec 26 23:19:12.466977268 (511h50m12.338060313s) [upload=449126]: Got next segment. Inline: false Dec 26 23:19:12.467022674 (511h50m12.338105709s) [upload=449126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:19:12.467028906 (511h50m12.338111951s) [upload=449126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:19:12.718488249 (511h50m12.589571284s) [seg_pos={0 0}, upload=449126]: Begin upload segment... Dec 26 23:19:12.718549737 (511h50m12.589632772s) [seg_pos={0 0}, upload=449126]: Done begin upload segment. Dec 26 23:19:12.718612126 (511h50m12.589695161s) [noise=true , address=45.91.116.6:4974 , piece_id=HBNDBNN5ESS3RPZBTZIWWPM4WSQC6HDAZLURIBRJTNCXBDZ5DYVQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.718695524 (511h50m12.589778569s) [noise=true , address=145.79.227.38:4977 , piece_id=P2R7BOAXHPQA2KV4F3VFXCVI42IQPF6PI4NP436SPTEN3OXMRAEA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.719430825 (511h50m12.59051386s) [noise=true , address=145.79.227.50:4373 , piece_id=ZU2D5OTWQDJICRQHBQBGN25E2PSCGJ7UI255IJEYEE365O6FRQLA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.719925256 (511h50m12.591008291s) [noise=true , address=145.79.227.51:4410 , piece_id=TBQOX3KZDEYCN4OJCIKDTRTYZG3U63GRNOR2VASS5YS6IS2Z3SMQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720102744 (511h50m12.591185779s) [noise=true , address=77.90.31.17:4773 , piece_id=3PEZ5BAJWFVCB327WFLTZ63FBKCUSDSHVLJBPAYPWRXHOVPZS2PA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720347370 (511h50m12.591430405s) [noise=true , address=145.79.227.5:4223 , piece_id=S3U2GOSLFCVL6NXVGUVSEUBKR7N746LNMJUAL2H7SAMLEISR2DEA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720454173 (511h50m12.591537208s) [noise=true , address=45.91.116.7:4011 , piece_id=T7GCLPLI74HDUC74Y26Q6ANIUVWYAQBZIMYPV44LLP5GQ7YIZH5Q, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720783159 (511h50m12.591866194s) [noise=true , address=145.79.227.43:4158 , piece_id=MZ4ZH44R7PPUPXGNQPAGVNPA2ARZPCO3Z6VALITWHIRLRBNAXXIQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720877379 (511h50m12.591960414s) [noise=true , address=145.79.227.45:4229 , piece_id=HZISC42PXKHME4QD4KWEXUZJNFDGJ2T4TXZ3TY6AGAOK6VC5KVWA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.721148295 (511h50m12.59223133s) [noise=true , address=77.90.31.16:4736 , piece_id=BIDIVS73X6IL4GAVPMG4IDGFBH5MICS2GXVBR422V6NLHGT7QCNA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.721424371 (511h50m12.592507406s) [noise=true , address=145.79.227.37:4565 , piece_id=OLCTEG2CU2X5IZVAWNKGRTO7G5EDN6UCC3UYIV3HP62OELSFX7GA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.721612629 (511h50m12.592695664s) [noise=true , address=145.79.227.11:4732 , piece_id=C2ARAE6EVIJN3BKIO73XSWS4ODN74GNNYNMIHFRW2JMGB2NAWQFA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.722022259 (511h50m12.593105304s) [noise=true , address=145.79.227.7:4295 , piece_id=3WJKNLCOXUSMWKV3ASL3VLZHD4MPCNZPTYEEFZNYO7SRGKKLNGRQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.722125967 (511h50m12.593209002s) [noise=true , address=77.90.31.13:4629 , piece_id=NUQHUVXGSUMMZT576QYY5YWYQ7RNGHOZC2MOAAZYY7RERNMS2TJA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.722612243 (511h50m12.593695278s) [noise=true , address=145.79.227.46:4601 , piece_id=HYME7ERIU5Z62M3K4JVYXHS3L7LPTCOPYSKHYQDZEOZ4CPJXHXEQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.722780423 (511h50m12.593863458s) [noise=true , address=45.91.116.13:4156 , piece_id=GGMBC45WPA67JJIMQYCOLZGMNKDDEEOWHDQWWJHZIVU33MVIPD2A, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:14.424582260 (511h50m14.295665305s) [upload=449126]: Next returned nil segment Dec 26 23:19:14.424595757 (511h50m14.295678802s) [upload=449126]: Waiting for error group managing segments... Dec 26 23:19:14.706470205 (511h50m14.57755324s) [noise=true , address=145.79.227.50:4373 , piece_id=ZU2D5OTWQDJICRQHBQBGN25E2PSCGJ7UI255IJEYEE365O6FRQLA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.712004413 (511h50m14.583087448s) [noise=true , address=145.79.227.51:4410 , piece_id=TBQOX3KZDEYCN4OJCIKDTRTYZG3U63GRNOR2VASS5YS6IS2Z3SMQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.712235883 (511h50m14.583318918s) [noise=true , address=145.79.227.38:4977 , piece_id=P2R7BOAXHPQA2KV4F3VFXCVI42IQPF6PI4NP436SPTEN3OXMRAEA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.717901832 (511h50m14.588984867s) [noise=true , address=145.79.227.11:4732 , piece_id=C2ARAE6EVIJN3BKIO73XSWS4ODN74GNNYNMIHFRW2JMGB2NAWQFA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.726128848 (511h50m14.597211893s) [noise=true , address=145.79.227.37:4565 , piece_id=OLCTEG2CU2X5IZVAWNKGRTO7G5EDN6UCC3UYIV3HP62OELSFX7GA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.744914238 (511h50m14.615997273s) [noise=true , address=145.79.227.45:4229 , piece_id=HZISC42PXKHME4QD4KWEXUZJNFDGJ2T4TXZ3TY6AGAOK6VC5KVWA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.746678788 (511h50m14.617761823s) [noise=true , address=145.79.227.5:4223 , piece_id=S3U2GOSLFCVL6NXVGUVSEUBKR7N746LNMJUAL2H7SAMLEISR2DEA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.762938268 (511h50m14.634021323s) [noise=true , address=145.79.227.46:4601 , piece_id=HYME7ERIU5Z62M3K4JVYXHS3L7LPTCOPYSKHYQDZEOZ4CPJXHXEQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.912235681 (511h50m14.783318726s) [noise=true , address=77.90.31.16:4736 , piece_id=BIDIVS73X6IL4GAVPMG4IDGFBH5MICS2GXVBR422V6NLHGT7QCNA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.121372597 (511h50m14.992455642s) [noise=true , address=45.91.116.6:4974 , piece_id=HBNDBNN5ESS3RPZBTZIWWPM4WSQC6HDAZLURIBRJTNCXBDZ5DYVQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.179091482 (511h50m15.050174527s) [noise=true , address=45.91.116.13:4156 , piece_id=GGMBC45WPA67JJIMQYCOLZGMNKDDEEOWHDQWWJHZIVU33MVIPD2A, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.643867068 (511h50m15.514950113s) [noise=true , address=77.90.31.17:4773 , piece_id=3PEZ5BAJWFVCB327WFLTZ63FBKCUSDSHVLJBPAYPWRXHOVPZS2PA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.762665928 (511h50m15.633748963s) [noise=true , address=77.90.31.13:4629 , piece_id=NUQHUVXGSUMMZT576QYY5YWYQ7RNGHOZC2MOAAZYY7RERNMS2TJA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.762688852 (511h50m15.633771897s) [seg_pos={0 0}, upload=449126]: Segment reached optimal threshold of 13 pieces. Dec 26 23:19:15.762917777 (511h50m15.634000822s) [noise=true , address=45.91.116.7:4011 , piece_id=T7GCLPLI74HDUC74Y26Q6ANIUVWYAQBZIMYPV44LLP5GQ7YIZH5Q, seg_pos={0 0}, upload=449126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 23:19:15.762933358 (511h50m15.634016393s) [noise=true , address=45.91.116.7:4011 , piece_id=T7GCLPLI74HDUC74Y26Q6ANIUVWYAQBZIMYPV44LLP5GQ7YIZH5Q, seg_pos={0 0}, upload=449126]: Long tail cancellation due to optimal threshold Dec 26 23:19:15.762959066 (511h50m15.634042111s) [noise=true , address=145.79.227.7:4295 , piece_id=3WJKNLCOXUSMWKV3ASL3VLZHD4MPCNZPTYEEFZNYO7SRGKKLNGRQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:41168->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41168->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41168->145.79.227.7:4295: use of closed network connection Dec 26 23:19:15.762968254 (511h50m15.634051289s) [noise=true , address=145.79.227.7:4295 , piece_id=3WJKNLCOXUSMWKV3ASL3VLZHD4MPCNZPTYEEFZNYO7SRGKKLNGRQ, seg_pos={0 0}, upload=449126]: Long tail cancellation due to optimal threshold Dec 26 23:19:15.763020493 (511h50m15.634103528s) [noise=true , address=145.79.227.43:4158 , piece_id=MZ4ZH44R7PPUPXGNQPAGVNPA2ARZPCO3Z6VALITWHIRLRBNAXXIQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 26 23:19:15.763026755 (511h50m15.63410979s) [noise=true , address=145.79.227.43:4158 , piece_id=MZ4ZH44R7PPUPXGNQPAGVNPA2ARZPCO3Z6VALITWHIRLRBNAXXIQ, seg_pos={0 0}, upload=449126]: Long tail cancellation due to optimal threshold Dec 26 23:19:15.763044008 (511h50m15.634127043s) [seg_pos={0 0}, upload=449126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:19:15.763113721 (511h50m15.634196756s) [upload=449126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:19:15.763118520 (511h50m15.634201565s) [upload=449126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:19:16.065024811 (511h50m15.936107866s) [upload=449126]: Done uploading segments... Dec 26 23:19:16.065046683 (511h50m15.936129718s) [upload=449126]: Upload finished. err: <nil> Duration: 24.489058425s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.932181859s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:20:38 UTC) orbiter warning: context deadline exceeded Duration: 5.932207828s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.8089082s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:19:51 UTC) orbiter warning: context deadline exceeded Duration: 5.808940372s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X....X.......X.......X..............................X.X.X] duration: 30.002142281s trace: https://jaeger.dev.storj.io/trace/136f097503400aab FAIL (2025-12-26 23:19:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: HBNDBNN5ESS3RPZBTZIWWPM4WSQC6HDAZLURIBRJTNCXBDZ5DYVQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: BIDIVS73X6IL4GAVPMG4IDGFBH5MICS2GXVBR422V6NLHGT7QCNA) context deadline exceeded ======LOGS====== 23:19:12.466635016 (511h50m12.337718061s) [upload=449126]: Starting upload Dec 26 23:19:12.466708205 (511h50m12.337791251s) [upload=449126]: Done starting upload Dec 26 23:19:12.466748342 (511h50m12.337831377s) [upload=449126]: Uploading segments... Dec 26 23:19:12.466977268 (511h50m12.338060313s) [upload=449126]: Got next segment. Inline: false Dec 26 23:19:12.467022674 (511h50m12.338105709s) [upload=449126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:19:12.467028906 (511h50m12.338111951s) [upload=449126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:19:12.718488249 (511h50m12.589571284s) [seg_pos={0 0}, upload=449126]: Begin upload segment... Dec 26 23:19:12.718549737 (511h50m12.589632772s) [seg_pos={0 0}, upload=449126]: Done begin upload segment. Dec 26 23:19:12.718612126 (511h50m12.589695161s) [noise=true , address=45.91.116.6:4974 , piece_id=HBNDBNN5ESS3RPZBTZIWWPM4WSQC6HDAZLURIBRJTNCXBDZ5DYVQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.718695524 (511h50m12.589778569s) [noise=true , address=145.79.227.38:4977 , piece_id=P2R7BOAXHPQA2KV4F3VFXCVI42IQPF6PI4NP436SPTEN3OXMRAEA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.719430825 (511h50m12.59051386s) [noise=true , address=145.79.227.50:4373 , piece_id=ZU2D5OTWQDJICRQHBQBGN25E2PSCGJ7UI255IJEYEE365O6FRQLA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.719925256 (511h50m12.591008291s) [noise=true , address=145.79.227.51:4410 , piece_id=TBQOX3KZDEYCN4OJCIKDTRTYZG3U63GRNOR2VASS5YS6IS2Z3SMQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720102744 (511h50m12.591185779s) [noise=true , address=77.90.31.17:4773 , piece_id=3PEZ5BAJWFVCB327WFLTZ63FBKCUSDSHVLJBPAYPWRXHOVPZS2PA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720347370 (511h50m12.591430405s) [noise=true , address=145.79.227.5:4223 , piece_id=S3U2GOSLFCVL6NXVGUVSEUBKR7N746LNMJUAL2H7SAMLEISR2DEA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720454173 (511h50m12.591537208s) [noise=true , address=45.91.116.7:4011 , piece_id=T7GCLPLI74HDUC74Y26Q6ANIUVWYAQBZIMYPV44LLP5GQ7YIZH5Q, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720783159 (511h50m12.591866194s) [noise=true , address=145.79.227.43:4158 , piece_id=MZ4ZH44R7PPUPXGNQPAGVNPA2ARZPCO3Z6VALITWHIRLRBNAXXIQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720877379 (511h50m12.591960414s) [noise=true , address=145.79.227.45:4229 , piece_id=HZISC42PXKHME4QD4KWEXUZJNFDGJ2T4TXZ3TY6AGAOK6VC5KVWA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.721148295 (511h50m12.59223133s) [noise=true , address=77.90.31.16:4736 , piece_id=BIDIVS73X6IL4GAVPMG4IDGFBH5MICS2GXVBR422V6NLHGT7QCNA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.721424371 (511h50m12.592507406s) [noise=true , address=145.79.227.37:4565 , piece_id=OLCTEG2CU2X5IZVAWNKGRTO7G5EDN6UCC3UYIV3HP62OELSFX7GA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.721612629 (511h50m12.592695664s) [noise=true , address=145.79.227.11:4732 , piece_id=C2ARAE6EVIJN3BKIO73XSWS4ODN74GNNYNMIHFRW2JMGB2NAWQFA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.722022259 (511h50m12.593105304s) [noise=true , address=145.79.227.7:4295 , piece_id=3WJKNLCOXUSMWKV3ASL3VLZHD4MPCNZPTYEEFZNYO7SRGKKLNGRQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.722125967 (511h50m12.593209002s) [noise=true , address=77.90.31.13:4629 , piece_id=NUQHUVXGSUMMZT576QYY5YWYQ7RNGHOZC2MOAAZYY7RERNMS2TJA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.722612243 (511h50m12.593695278s) [noise=true , address=145.79.227.46:4601 , piece_id=HYME7ERIU5Z62M3K4JVYXHS3L7LPTCOPYSKHYQDZEOZ4CPJXHXEQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.722780423 (511h50m12.593863458s) [noise=true , address=45.91.116.13:4156 , piece_id=GGMBC45WPA67JJIMQYCOLZGMNKDDEEOWHDQWWJHZIVU33MVIPD2A, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:14.424582260 (511h50m14.295665305s) [upload=449126]: Next returned nil segment Dec 26 23:19:14.424595757 (511h50m14.295678802s) [upload=449126]: Waiting for error group managing segments... Dec 26 23:19:14.706470205 (511h50m14.57755324s) [noise=true , address=145.79.227.50:4373 , piece_id=ZU2D5OTWQDJICRQHBQBGN25E2PSCGJ7UI255IJEYEE365O6FRQLA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.712004413 (511h50m14.583087448s) [noise=true , address=145.79.227.51:4410 , piece_id=TBQOX3KZDEYCN4OJCIKDTRTYZG3U63GRNOR2VASS5YS6IS2Z3SMQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.712235883 (511h50m14.583318918s) [noise=true , address=145.79.227.38:4977 , piece_id=P2R7BOAXHPQA2KV4F3VFXCVI42IQPF6PI4NP436SPTEN3OXMRAEA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.717901832 (511h50m14.588984867s) [noise=true , address=145.79.227.11:4732 , piece_id=C2ARAE6EVIJN3BKIO73XSWS4ODN74GNNYNMIHFRW2JMGB2NAWQFA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.726128848 (511h50m14.597211893s) [noise=true , address=145.79.227.37:4565 , piece_id=OLCTEG2CU2X5IZVAWNKGRTO7G5EDN6UCC3UYIV3HP62OELSFX7GA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.744914238 (511h50m14.615997273s) [noise=true , address=145.79.227.45:4229 , piece_id=HZISC42PXKHME4QD4KWEXUZJNFDGJ2T4TXZ3TY6AGAOK6VC5KVWA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.746678788 (511h50m14.617761823s) [noise=true , address=145.79.227.5:4223 , piece_id=S3U2GOSLFCVL6NXVGUVSEUBKR7N746LNMJUAL2H7SAMLEISR2DEA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.762938268 (511h50m14.634021323s) [noise=true , address=145.79.227.46:4601 , piece_id=HYME7ERIU5Z62M3K4JVYXHS3L7LPTCOPYSKHYQDZEOZ4CPJXHXEQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.912235681 (511h50m14.783318726s) [noise=true , address=77.90.31.16:4736 , piece_id=BIDIVS73X6IL4GAVPMG4IDGFBH5MICS2GXVBR422V6NLHGT7QCNA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.121372597 (511h50m14.992455642s) [noise=true , address=45.91.116.6:4974 , piece_id=HBNDBNN5ESS3RPZBTZIWWPM4WSQC6HDAZLURIBRJTNCXBDZ5DYVQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.179091482 (511h50m15.050174527s) [noise=true , address=45.91.116.13:4156 , piece_id=GGMBC45WPA67JJIMQYCOLZGMNKDDEEOWHDQWWJHZIVU33MVIPD2A, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.643867068 (511h50m15.514950113s) [noise=true , address=77.90.31.17:4773 , piece_id=3PEZ5BAJWFVCB327WFLTZ63FBKCUSDSHVLJBPAYPWRXHOVPZS2PA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.762665928 (511h50m15.633748963s) [noise=true , address=77.90.31.13:4629 , piece_id=NUQHUVXGSUMMZT576QYY5YWYQ7RNGHOZC2MOAAZYY7RERNMS2TJA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.762688852 (511h50m15.633771897s) [seg_pos={0 0}, upload=449126]: Segment reached optimal threshold of 13 pieces. Dec 26 23:19:15.762917777 (511h50m15.634000822s) [noise=true , address=45.91.116.7:4011 , piece_id=T7GCLPLI74HDUC74Y26Q6ANIUVWYAQBZIMYPV44LLP5GQ7YIZH5Q, seg_pos={0 0}, upload=449126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 23:19:15.762933358 (511h50m15.634016393s) [noise=true , address=45.91.116.7:4011 , piece_id=T7GCLPLI74HDUC74Y26Q6ANIUVWYAQBZIMYPV44LLP5GQ7YIZH5Q, seg_pos={0 0}, upload=449126]: Long tail cancellation due to optimal threshold Dec 26 23:19:15.762959066 (511h50m15.634042111s) [noise=true , address=145.79.227.7:4295 , piece_id=3WJKNLCOXUSMWKV3ASL3VLZHD4MPCNZPTYEEFZNYO7SRGKKLNGRQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:41168->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41168->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41168->145.79.227.7:4295: use of closed network connection Dec 26 23:19:15.762968254 (511h50m15.634051289s) [noise=true , address=145.79.227.7:4295 , piece_id=3WJKNLCOXUSMWKV3ASL3VLZHD4MPCNZPTYEEFZNYO7SRGKKLNGRQ, seg_pos={0 0}, upload=449126]: Long tail cancellation due to optimal threshold Dec 26 23:19:15.763020493 (511h50m15.634103528s) [noise=true , address=145.79.227.43:4158 , piece_id=MZ4ZH44R7PPUPXGNQPAGVNPA2ARZPCO3Z6VALITWHIRLRBNAXXIQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 26 23:19:15.763026755 (511h50m15.63410979s) [noise=true , address=145.79.227.43:4158 , piece_id=MZ4ZH44R7PPUPXGNQPAGVNPA2ARZPCO3Z6VALITWHIRLRBNAXXIQ, seg_pos={0 0}, upload=449126]: Long tail cancellation due to optimal threshold Dec 26 23:19:15.763044008 (511h50m15.634127043s) [seg_pos={0 0}, upload=449126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:19:15.763113721 (511h50m15.634196756s) [upload=449126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:19:15.763118520 (511h50m15.634201565s) [upload=449126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:19:16.065024811 (511h50m15.936107866s) [upload=449126]: Done uploading segments... Dec 26 23:19:16.065046683 (511h50m15.936129718s) [upload=449126]: Upload finished. err: <nil> Duration: 30.002165044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X....X.......X.......X..............................X.X.] duration: 6.062299474s trace: https://jaeger.dev.storj.io/trace/671df6d0e83b79ba FAIL (2025-12-26 23:18:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: QF6ZAHTPCPNBDETIZ23XLHVXNCPCUHYLKYQPB3E6NTY47G5JCZCA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: LOCUEZQOQ6OH7KCRXL4OJVEYIYAEAEJLY6XJP3DGBNOZBIRVVPYA) context deadline exceeded ======LOGS====== Dec 26 23:17:44.491436204 (511h48m44.362519239s) [upload=449107]: Starting upload Dec 26 23:17:44.491497711 (511h48m44.362580746s) [upload=449107]: Done starting upload Dec 26 23:17:44.491591670 (511h48m44.362674705s) [upload=449107]: Uploading segments... Dec 26 23:17:44.491687372 (511h48m44.362770407s) [upload=449107]: Got next segment. Inline: false Dec 26 23:17:44.491738199 (511h48m44.362821234s) [upload=449107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:17:44.491745623 (511h48m44.362828658s) [upload=449107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:17:44.742342985 (511h48m44.61342603s) [seg_pos={0 0}, upload=449107]: Begin upload segment... Dec 26 23:17:44.742384525 (511h48m44.61346757s) [seg_pos={0 0}, upload=449107]: Done begin upload segment. Dec 26 23:17:44.742411195 (511h48m44.61349423s) [noise=true , address=45.91.116.7:4011 , piece_id=KCIYMORIJFMSSXTI4DOEJI7GXTCOR5TFILRMYFIAIZVV7PJVBVIA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.742415975 (511h48m44.61349901s) [noise=true , address=145.79.227.5:4221 , piece_id=MALZPJVFO6ICWSHH3AZLJZY5H67VKIYYLPYN7G74ZVAQR62KTBSA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.742821496 (511h48m44.613904541s) [noise=true , address=45.91.116.6:4976 , piece_id=LOCUEZQOQ6OH7KCRXL4OJVEYIYAEAEJLY6XJP3DGBNOZBIRVVPYA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.743653751 (511h48m44.614736786s) [noise=true , address=145.79.227.48:4301 , piece_id=CW4SM5SZPCLFJ3AENH6QNSZEKAD7ZJNO7WG4JSV6IHNBKGX7A2PA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.744041980 (511h48m44.615125015s) [noise=true , address=45.91.116.9:4261 , piece_id=JVDKOZXH6HVPD7BSGKARQTEMQADKD2J7AS2JJAVGJ45IKKPGBBLA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.744649828 (511h48m44.615732863s) [noise=true , address=45.91.116.5:4299 , piece_id=AJ3422X3RNUAPJ6VVNY75LQMV6CHFLI45P6TCW36M6IGMBONFWKQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.744813028 (511h48m44.615896073s) [noise=true , address=145.79.227.39:4014 , piece_id=6CBYURI5XNZ7Z4XOM47LEUOONRRKEOIAZUG2H4JY4KDRYG4ANFTQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.744985236 (511h48m44.616068271s) [noise=true , address=145.79.227.37:4565 , piece_id=OQGE7BPBUTZ7WFMAHB57FLANU4OFNGAMUV6V2CKNYVZPHPCYTMMA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.745843140 (511h48m44.616926175s) [noise=true , address=145.79.227.45:4229 , piece_id=ZGGG32VDXT76S7P2CVMNIR5LU4IQ6XPKC3K3CRCAJRSO6HTUME6Q, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.745958499 (511h48m44.617041545s) [noise=true , address=145.79.227.42:4122 , piece_id=FLJF66ZEKF7GYHHQWLUMMHQHWXNIU6BWSBWP2G4J6PODGGHCO26Q, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.747158485 (511h48m44.61824154s) [noise=true , address=45.91.116.4:4937 , piece_id=ZZSI3MJWIOG6FEHB3A6TJ34RK2LOLXEG7UAAQLYMFB7LDZWUJDIQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.748134744 (511h48m44.619217789s) [noise=true , address=145.79.227.46:4602 , piece_id=S4SNS3KE2SAIW54QMEH6UUMXRKQET3TUWXCU5R77ZRNCIIYG4JUA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.749173332 (511h48m44.620256377s) [noise=true , address=145.79.227.12:4765 , piece_id=QCDW2SVVZWPIUHUPKFP7HYPVC2IMWTIRBI5FR2INT6QIPAT235HQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.751251518 (511h48m44.622334553s) [noise=true , address=45.91.116.14:4190 , piece_id=QF6ZAHTPCPNBDETIZ23XLHVXNCPCUHYLKYQPB3E6NTY47G5JCZCA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.752908283 (511h48m44.623991318s) [noise=true , address=145.79.227.13:4804 , piece_id=PPQJUBGJPENZOSVAYCV2VK26NK3BZOWOGB47DCBFQDFMFDWQ4JOQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.757383205 (511h48m44.62846626s) [noise=true , address=145.79.227.11:4732 , piece_id=CSCOBQMIUA5VTK7FVHXBFTLMHIV4BFQ5ZT4WEFT6Y57XOIF7F6HQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:46.459493209 (511h48m46.330576244s) [upload=449107]: Next returned nil segment Dec 26 23:17:46.459507356 (511h48m46.330590391s) [upload=449107]: Waiting for error group managing segments... Dec 26 23:17:46.751409548 (511h48m46.622492594s) [noise=true , address=145.79.227.48:4301 , piece_id=CW4SM5SZPCLFJ3AENH6QNSZEKAD7ZJNO7WG4JSV6IHNBKGX7A2PA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.761303899 (511h48m46.632386954s) [noise=true , address=145.79.227.45:4229 , piece_id=ZGGG32VDXT76S7P2CVMNIR5LU4IQ6XPKC3K3CRCAJRSO6HTUME6Q, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.761498961 (511h48m46.632582006s) [noise=true , address=145.79.227.39:4014 , piece_id=6CBYURI5XNZ7Z4XOM47LEUOONRRKEOIAZUG2H4JY4KDRYG4ANFTQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.765415678 (511h48m46.636498713s) [noise=true , address=145.79.227.5:4221 , piece_id=MALZPJVFO6ICWSHH3AZLJZY5H67VKIYYLPYN7G74ZVAQR62KTBSA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.765491764 (511h48m46.636574799s) [noise=true , address=145.79.227.37:4565 , piece_id=OQGE7BPBUTZ7WFMAHB57FLANU4OFNGAMUV6V2CKNYVZPHPCYTMMA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.782774852 (511h48m46.653857887s) [noise=true , address=145.79.227.42:4122 , piece_id=FLJF66ZEKF7GYHHQWLUMMHQHWXNIU6BWSBWP2G4J6PODGGHCO26Q, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.783704623 (511h48m46.654787658s) [noise=true , address=145.79.227.13:4804 , piece_id=PPQJUBGJPENZOSVAYCV2VK26NK3BZOWOGB47DCBFQDFMFDWQ4JOQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.783994234 (511h48m46.655077269s) [noise=true , address=145.79.227.11:4732 , piece_id=CSCOBQMIUA5VTK7FVHXBFTLMHIV4BFQ5ZT4WEFT6Y57XOIF7F6HQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.787077045 (511h48m46.65816008s) [noise=true , address=145.79.227.46:4602 , piece_id=S4SNS3KE2SAIW54QMEH6UUMXRKQET3TUWXCU5R77ZRNCIIYG4JUA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.800716205 (511h48m46.67179924s) [noise=true , address=145.79.227.12:4765 , piece_id=QCDW2SVVZWPIUHUPKFP7HYPVC2IMWTIRBI5FR2INT6QIPAT235HQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:47.009597595 (511h48m46.88068063s) [noise=true , address=45.91.116.6:4976 , piece_id=LOCUEZQOQ6OH7KCRXL4OJVEYIYAEAEJLY6XJP3DGBNOZBIRVVPYA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:47.040843763 (511h48m46.911926808s) [noise=true , address=45.91.116.5:4299 , piece_id=AJ3422X3RNUAPJ6VVNY75LQMV6CHFLI45P6TCW36M6IGMBONFWKQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:47.046454385 (511h48m46.91753743s) [noise=true , address=45.91.116.14:4190 , piece_id=QF6ZAHTPCPNBDETIZ23XLHVXNCPCUHYLKYQPB3E6NTY47G5JCZCA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:47.046471699 (511h48m46.917554734s) [seg_pos={0 0}, upload=449107]: Segment reached optimal threshold of 13 pieces. Dec 26 23:17:47.046727105 (511h48m46.91781015s) [noise=true , address=45.91.116.4:4937 , piece_id=ZZSI3MJWIOG6FEHB3A6TJ34RK2LOLXEG7UAAQLYMFB7LDZWUJDIQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 23:17:47.046736051 (511h48m46.917819086s) [noise=true , address=45.91.116.4:4937 , piece_id=ZZSI3MJWIOG6FEHB3A6TJ34RK2LOLXEG7UAAQLYMFB7LDZWUJDIQ, seg_pos={0 0}, upload=449107]: Long tail cancellation due to optimal threshold Dec 26 23:17:47.046969005 (511h48m46.91805204s) [noise=true , address=45.91.116.9:4261 , piece_id=JVDKOZXH6HVPD7BSGKARQTEMQADKD2J7AS2JJAVGJ45IKKPGBBLA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 23:17:47.046975277 (511h48m46.918058312s) [noise=true , address=45.91.116.9:4261 , piece_id=JVDKOZXH6HVPD7BSGKARQTEMQADKD2J7AS2JJAVGJ45IKKPGBBLA, seg_pos={0 0}, upload=449107]: Long tail cancellation due to optimal threshold Dec 26 23:17:47.047225994 (511h48m46.918309029s) [noise=true , address=45.91.116.7:4011 , piece_id=KCIYMORIJFMSSXTI4DOEJI7GXTCOR5TFILRMYFIAIZVV7PJVBVIA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 23:17:47.047236405 (511h48m46.91831945s) [noise=true , address=45.91.116.7:4011 , piece_id=KCIYMORIJFMSSXTI4DOEJI7GXTCOR5TFILRMYFIAIZVV7PJVBVIA, seg_pos={0 0}, upload=449107]: Long tail cancellation due to optimal threshold Dec 26 23:17:47.047259197 (511h48m46.918342233s) [seg_pos={0 0}, upload=449107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:17:47.047368827 (511h48m46.918451862s) [upload=449107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:17:47.047374959 (511h48m46.918458004s) [upload=449107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:17:47.350840189 (511h48m47.221923234s) [upload=449107]: Done uploading segments... Dec 26 23:17:47.350858774 (511h48m47.221941809s) [upload=449107]: Upload finished. err: <nil> Duration: 6.062330021s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.836973718s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:18:41 UTC) orbiter warning: context deadline exceeded Duration: 5.836997984s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X....X.......X.......X..............................X.X] duration: 30.001772348s trace: https://jaeger.dev.storj.io/trace/671df6d0e83b79ba FAIL (2025-12-26 23:18:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: QF6ZAHTPCPNBDETIZ23XLHVXNCPCUHYLKYQPB3E6NTY47G5JCZCA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: LOCUEZQOQ6OH7KCRXL4OJVEYIYAEAEJLY6XJP3DGBNOZBIRVVPYA) context deadline exceeded ======LOGS====== Dec 26 23:17:44.491436204 (511h48m44.362519239s) [upload=449107]: Starting upload Dec 26 23:17:44.491497711 (511h48m44.362580746s) [upload=449107]: Done starting upload Dec 26 23:17:44.491591670 (511h48m44.362674705s) [upload=449107]: Uploading segments... Dec 26 23:17:44.491687372 (511h48m44.362770407s) [upload=449107]: Got next segment. Inline: false Dec 26 23:17:44.491738199 (511h48m44.362821234s) [upload=449107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:17:44.491745623 (511h48m44.362828658s) [upload=449107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:17:44.742342985 (511h48m44.61342603s) [seg_pos={0 0}, upload=449107]: Begin upload segment... Dec 26 23:17:44.742384525 (511h48m44.61346757s) [seg_pos={0 0}, upload=449107]: Done begin upload segment. Dec 26 23:17:44.742411195 (511h48m44.61349423s) [noise=true , address=45.91.116.7:4011 , piece_id=KCIYMORIJFMSSXTI4DOEJI7GXTCOR5TFILRMYFIAIZVV7PJVBVIA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.742415975 (511h48m44.61349901s) [noise=true , address=145.79.227.5:4221 , piece_id=MALZPJVFO6ICWSHH3AZLJZY5H67VKIYYLPYN7G74ZVAQR62KTBSA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.742821496 (511h48m44.613904541s) [noise=true , address=45.91.116.6:4976 , piece_id=LOCUEZQOQ6OH7KCRXL4OJVEYIYAEAEJLY6XJP3DGBNOZBIRVVPYA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.743653751 (511h48m44.614736786s) [noise=true , address=145.79.227.48:4301 , piece_id=CW4SM5SZPCLFJ3AENH6QNSZEKAD7ZJNO7WG4JSV6IHNBKGX7A2PA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.744041980 (511h48m44.615125015s) [noise=true , address=45.91.116.9:4261 , piece_id=JVDKOZXH6HVPD7BSGKARQTEMQADKD2J7AS2JJAVGJ45IKKPGBBLA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.744649828 (511h48m44.615732863s) [noise=true , address=45.91.116.5:4299 , piece_id=AJ3422X3RNUAPJ6VVNY75LQMV6CHFLI45P6TCW36M6IGMBONFWKQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.744813028 (511h48m44.615896073s) [noise=true , address=145.79.227.39:4014 , piece_id=6CBYURI5XNZ7Z4XOM47LEUOONRRKEOIAZUG2H4JY4KDRYG4ANFTQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.744985236 (511h48m44.616068271s) [noise=true , address=145.79.227.37:4565 , piece_id=OQGE7BPBUTZ7WFMAHB57FLANU4OFNGAMUV6V2CKNYVZPHPCYTMMA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.745843140 (511h48m44.616926175s) [noise=true , address=145.79.227.45:4229 , piece_id=ZGGG32VDXT76S7P2CVMNIR5LU4IQ6XPKC3K3CRCAJRSO6HTUME6Q, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.745958499 (511h48m44.617041545s) [noise=true , address=145.79.227.42:4122 , piece_id=FLJF66ZEKF7GYHHQWLUMMHQHWXNIU6BWSBWP2G4J6PODGGHCO26Q, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.747158485 (511h48m44.61824154s) [noise=true , address=45.91.116.4:4937 , piece_id=ZZSI3MJWIOG6FEHB3A6TJ34RK2LOLXEG7UAAQLYMFB7LDZWUJDIQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.748134744 (511h48m44.619217789s) [noise=true , address=145.79.227.46:4602 , piece_id=S4SNS3KE2SAIW54QMEH6UUMXRKQET3TUWXCU5R77ZRNCIIYG4JUA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.749173332 (511h48m44.620256377s) [noise=true , address=145.79.227.12:4765 , piece_id=QCDW2SVVZWPIUHUPKFP7HYPVC2IMWTIRBI5FR2INT6QIPAT235HQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.751251518 (511h48m44.622334553s) [noise=true , address=45.91.116.14:4190 , piece_id=QF6ZAHTPCPNBDETIZ23XLHVXNCPCUHYLKYQPB3E6NTY47G5JCZCA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.752908283 (511h48m44.623991318s) [noise=true , address=145.79.227.13:4804 , piece_id=PPQJUBGJPENZOSVAYCV2VK26NK3BZOWOGB47DCBFQDFMFDWQ4JOQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.757383205 (511h48m44.62846626s) [noise=true , address=145.79.227.11:4732 , piece_id=CSCOBQMIUA5VTK7FVHXBFTLMHIV4BFQ5ZT4WEFT6Y57XOIF7F6HQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:46.459493209 (511h48m46.330576244s) [upload=449107]: Next returned nil segment Dec 26 23:17:46.459507356 (511h48m46.330590391s) [upload=449107]: Waiting for error group managing segments... Dec 26 23:17:46.751409548 (511h48m46.622492594s) [noise=true , address=145.79.227.48:4301 , piece_id=CW4SM5SZPCLFJ3AENH6QNSZEKAD7ZJNO7WG4JSV6IHNBKGX7A2PA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.761303899 (511h48m46.632386954s) [noise=true , address=145.79.227.45:4229 , piece_id=ZGGG32VDXT76S7P2CVMNIR5LU4IQ6XPKC3K3CRCAJRSO6HTUME6Q, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.761498961 (511h48m46.632582006s) [noise=true , address=145.79.227.39:4014 , piece_id=6CBYURI5XNZ7Z4XOM47LEUOONRRKEOIAZUG2H4JY4KDRYG4ANFTQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.765415678 (511h48m46.636498713s) [noise=true , address=145.79.227.5:4221 , piece_id=MALZPJVFO6ICWSHH3AZLJZY5H67VKIYYLPYN7G74ZVAQR62KTBSA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.765491764 (511h48m46.636574799s) [noise=true , address=145.79.227.37:4565 , piece_id=OQGE7BPBUTZ7WFMAHB57FLANU4OFNGAMUV6V2CKNYVZPHPCYTMMA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.782774852 (511h48m46.653857887s) [noise=true , address=145.79.227.42:4122 , piece_id=FLJF66ZEKF7GYHHQWLUMMHQHWXNIU6BWSBWP2G4J6PODGGHCO26Q, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.783704623 (511h48m46.654787658s) [noise=true , address=145.79.227.13:4804 , piece_id=PPQJUBGJPENZOSVAYCV2VK26NK3BZOWOGB47DCBFQDFMFDWQ4JOQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.783994234 (511h48m46.655077269s) [noise=true , address=145.79.227.11:4732 , piece_id=CSCOBQMIUA5VTK7FVHXBFTLMHIV4BFQ5ZT4WEFT6Y57XOIF7F6HQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.787077045 (511h48m46.65816008s) [noise=true , address=145.79.227.46:4602 , piece_id=S4SNS3KE2SAIW54QMEH6UUMXRKQET3TUWXCU5R77ZRNCIIYG4JUA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.800716205 (511h48m46.67179924s) [noise=true , address=145.79.227.12:4765 , piece_id=QCDW2SVVZWPIUHUPKFP7HYPVC2IMWTIRBI5FR2INT6QIPAT235HQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:47.009597595 (511h48m46.88068063s) [noise=true , address=45.91.116.6:4976 , piece_id=LOCUEZQOQ6OH7KCRXL4OJVEYIYAEAEJLY6XJP3DGBNOZBIRVVPYA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:47.040843763 (511h48m46.911926808s) [noise=true , address=45.91.116.5:4299 , piece_id=AJ3422X3RNUAPJ6VVNY75LQMV6CHFLI45P6TCW36M6IGMBONFWKQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:47.046454385 (511h48m46.91753743s) [noise=true , address=45.91.116.14:4190 , piece_id=QF6ZAHTPCPNBDETIZ23XLHVXNCPCUHYLKYQPB3E6NTY47G5JCZCA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:47.046471699 (511h48m46.917554734s) [seg_pos={0 0}, upload=449107]: Segment reached optimal threshold of 13 pieces. Dec 26 23:17:47.046727105 (511h48m46.91781015s) [noise=true , address=45.91.116.4:4937 , piece_id=ZZSI3MJWIOG6FEHB3A6TJ34RK2LOLXEG7UAAQLYMFB7LDZWUJDIQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 23:17:47.046736051 (511h48m46.917819086s) [noise=true , address=45.91.116.4:4937 , piece_id=ZZSI3MJWIOG6FEHB3A6TJ34RK2LOLXEG7UAAQLYMFB7LDZWUJDIQ, seg_pos={0 0}, upload=449107]: Long tail cancellation due to optimal threshold Dec 26 23:17:47.046969005 (511h48m46.91805204s) [noise=true , address=45.91.116.9:4261 , piece_id=JVDKOZXH6HVPD7BSGKARQTEMQADKD2J7AS2JJAVGJ45IKKPGBBLA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 23:17:47.046975277 (511h48m46.918058312s) [noise=true , address=45.91.116.9:4261 , piece_id=JVDKOZXH6HVPD7BSGKARQTEMQADKD2J7AS2JJAVGJ45IKKPGBBLA, seg_pos={0 0}, upload=449107]: Long tail cancellation due to optimal threshold Dec 26 23:17:47.047225994 (511h48m46.918309029s) [noise=true , address=45.91.116.7:4011 , piece_id=KCIYMORIJFMSSXTI4DOEJI7GXTCOR5TFILRMYFIAIZVV7PJVBVIA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 23:17:47.047236405 (511h48m46.91831945s) [noise=true , address=45.91.116.7:4011 , piece_id=KCIYMORIJFMSSXTI4DOEJI7GXTCOR5TFILRMYFIAIZVV7PJVBVIA, seg_pos={0 0}, upload=449107]: Long tail cancellation due to optimal threshold Dec 26 23:17:47.047259197 (511h48m46.918342233s) [seg_pos={0 0}, upload=449107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:17:47.047368827 (511h48m46.918451862s) [upload=449107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:17:47.047374959 (511h48m46.918458004s) [upload=449107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:17:47.350840189 (511h48m47.221923234s) [upload=449107]: Done uploading segments... Dec 26 23:17:47.350858774 (511h48m47.221941809s) [upload=449107]: Upload finished. err: <nil> Duration: 30.00179973s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.856806568s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:17:27 UTC) orbiter warning: context deadline exceeded Duration: 5.85683412s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.235429477s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:17:02 UTC) orbiter warning: context deadline exceeded Duration: 6.235454556s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.....X....X.......X.......X..............................X.] duration: 6.172218754s trace: https://jaeger.dev.storj.io/trace/130eb89f4e53f1e0 FAIL (2025-12-26 23:16:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: BJHWKNTZFP7FJCGSDW5PK6QIQIFAWPM2YUHDIK4ULBKOCCRVPFIQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 36QPCOF5BYCWH3FWKFYGTQYNYPITIHJYHEYHNUN74EV3CMUGHR7A) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: SDSXPMCEU34QOVN35VFPVAAHUVCMWDS2SWUMAMCBX3ZZGCFLXEXA) context deadline exceeded ======LOGS====== 449086]: Starting upload Dec 26 23:16:03.827516883 (511h47m3.698599928s) [upload=449086]: Done starting upload Dec 26 23:16:03.827664724 (511h47m3.698747769s) [upload=449086]: Uploading segments... Dec 26 23:16:03.827810051 (511h47m3.698893086s) [upload=449086]: Got next segment. Inline: false Dec 26 23:16:03.827849877 (511h47m3.698932922s) [upload=449086]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:16:03.827854716 (511h47m3.698937751s) [upload=449086]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:16:04.076311273 (511h47m3.947394328s) [seg_pos={0 0}, upload=449086]: Begin upload segment... Dec 26 23:16:04.076387258 (511h47m3.947470303s) [seg_pos={0 0}, upload=449086]: Done begin upload segment. Dec 26 23:16:04.076424929 (511h47m3.947507974s) [noise=true , address=45.91.116.7:4011 , piece_id=7W5ARDKVKREZSW7BDLI2NUQAMP2MTVKBVTNORKXRV44N5R3SCS4A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.077198041 (511h47m3.948281086s) [noise=true , address=77.90.31.16:4735 , piece_id=EDEVQWV4OXGD2I4I4OQ3OTU43UNPPNZGXC3KHONSLXTRGGGIXFMQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.077396269 (511h47m3.948479304s) [noise=true , address=145.79.227.51:4409 , piece_id=NW3YMT4WJOTESBG4VLZMA5QLC6QBSMFQ6RUE7QJHRRQSDHUCTPIQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.077886262 (511h47m3.948969297s) [noise=true , address=45.91.116.5:4298 , piece_id=J4VPBFSGIYAFU3QJRNVK26XOPBVXCZZUHOTPRXITLNDXEFRVLH4A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.078070983 (511h47m3.949154019s) [noise=true , address=145.79.227.50:4373 , piece_id=PYWDDOIZMFWSODZHE2INI3VGFSUOCGSIK4MCTVET2VC563PJVLWA, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.078379481 (511h47m3.949462526s) [noise=true , address=145.79.227.52:4446 , piece_id=HIWZ5KJ5NS6E4OEU5THWMYXRSFGBLR5XRQTJQQKRUR6URZKXDVGQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.078766578 (511h47m3.949849623s) [noise=true , address=45.91.116.11:4119 , piece_id=SDSXPMCEU34QOVN35VFPVAAHUVCMWDS2SWUMAMCBX3ZZGCFLXEXA, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.079195546 (511h47m3.950278581s) [noise=true , address=145.79.227.49:4338 , piece_id=XUC5VF2KVI2M3WSXME6TTPJDTZD5WPWOLOJEQDD67W2QKDDW345A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.079657595 (511h47m3.95074064s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VRI5ZGK2XLDLT6MBPJUXXTF4JGLVBQFAALJUDSRL4YXAJKQ5ZA, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.080066523 (511h47m3.951149558s) [noise=true , address=145.79.227.47:4267 , piece_id=ANZOJFHTGGWU3HE24OHZ7YWF5M6ONZBVQVYFTWRDN6PWMUS2IQNQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.080149692 (511h47m3.951232727s) [noise=true , address=45.91.116.4:4939 , piece_id=2LDMFLULUFKDEM7FRFMJP5STS6KDD6XAFCQJE7BFLCOT26ELAFAA, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.080585060 (511h47m3.951668106s) [noise=true , address=145.79.227.38:4978 , piece_id=3L5EOFRYYHYUGUAXH6LFCB7JLDNF3ASWMVFBCDF7Z7NMD3INHU5A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.080969733 (511h47m3.952052778s) [noise=true , address=45.91.116.12:4228 , piece_id=BJHWKNTZFP7FJCGSDW5PK6QIQIFAWPM2YUHDIK4ULBKOCCRVPFIQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.081353323 (511h47m3.952436368s) [noise=true , address=145.79.227.37:4567 , piece_id=PSOXFGFGPNSFWQRP2C3WOBI6ERTLA4K4AZNKS2WB2DKPAK6DWOPQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.081787120 (511h47m3.952870165s) [noise=true , address=45.91.116.8:4046 , piece_id=36QPCOF5BYCWH3FWKFYGTQYNYPITIHJYHEYHNUN74EV3CMUGHR7A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.081961282 (511h47m3.953044327s) [noise=true , address=45.91.116.9:4263 , piece_id=GCFI63PMDS2MTX6WHFW3MKFPOTXAWVI6HQVWNVRUBEXZ6UVFUQNQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:05.800268217 (511h47m5.671351252s) [upload=449086]: Next returned nil segment Dec 26 23:16:05.800289818 (511h47m5.671372863s) [upload=449086]: Waiting for error group managing segments... Dec 26 23:16:06.086124055 (511h47m5.95720709s) [noise=true , address=145.79.227.50:4373 , piece_id=PYWDDOIZMFWSODZHE2INI3VGFSUOCGSIK4MCTVET2VC563PJVLWA, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.095285099 (511h47m5.966368135s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VRI5ZGK2XLDLT6MBPJUXXTF4JGLVBQFAALJUDSRL4YXAJKQ5ZA, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.096266187 (511h47m5.967349233s) [noise=true , address=145.79.227.51:4409 , piece_id=NW3YMT4WJOTESBG4VLZMA5QLC6QBSMFQ6RUE7QJHRRQSDHUCTPIQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.099479436 (511h47m5.970562471s) [noise=true , address=145.79.227.52:4446 , piece_id=HIWZ5KJ5NS6E4OEU5THWMYXRSFGBLR5XRQTJQQKRUR6URZKXDVGQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.099654379 (511h47m5.970737444s) [noise=true , address=145.79.227.37:4567 , piece_id=PSOXFGFGPNSFWQRP2C3WOBI6ERTLA4K4AZNKS2WB2DKPAK6DWOPQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.100010197 (511h47m5.971093242s) [noise=true , address=145.79.227.49:4338 , piece_id=XUC5VF2KVI2M3WSXME6TTPJDTZD5WPWOLOJEQDD67W2QKDDW345A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.102804658 (511h47m5.973887693s) [noise=true , address=145.79.227.38:4978 , piece_id=3L5EOFRYYHYUGUAXH6LFCB7JLDNF3ASWMVFBCDF7Z7NMD3INHU5A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.114909597 (511h47m5.985992642s) [noise=true , address=145.79.227.47:4267 , piece_id=ANZOJFHTGGWU3HE24OHZ7YWF5M6ONZBVQVYFTWRDN6PWMUS2IQNQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.327932642 (511h47m6.199015677s) [noise=true , address=45.91.116.12:4228 , piece_id=BJHWKNTZFP7FJCGSDW5PK6QIQIFAWPM2YUHDIK4ULBKOCCRVPFIQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.356727242 (511h47m6.227810287s) [noise=true , address=45.91.116.4:4939 , piece_id=2LDMFLULUFKDEM7FRFMJP5STS6KDD6XAFCQJE7BFLCOT26ELAFAA, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.361363350 (511h47m6.232446395s) [noise=true , address=45.91.116.8:4046 , piece_id=36QPCOF5BYCWH3FWKFYGTQYNYPITIHJYHEYHNUN74EV3CMUGHR7A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.388538935 (511h47m6.25962198s) [noise=true , address=45.91.116.5:4298 , piece_id=J4VPBFSGIYAFU3QJRNVK26XOPBVXCZZUHOTPRXITLNDXEFRVLH4A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.439308448 (511h47m6.310391483s) [noise=true , address=45.91.116.11:4119 , piece_id=SDSXPMCEU34QOVN35VFPVAAHUVCMWDS2SWUMAMCBX3ZZGCFLXEXA, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.439329067 (511h47m6.310412102s) [seg_pos={0 0}, upload=449086]: Segment reached optimal threshold of 13 pieces. Dec 26 23:16:06.439539608 (511h47m6.310622643s) [noise=true , address=45.91.116.9:4263 , piece_id=GCFI63PMDS2MTX6WHFW3MKFPOTXAWVI6HQVWNVRUBEXZ6UVFUQNQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 23:16:06.439545950 (511h47m6.310628985s) [noise=true , address=45.91.116.9:4263 , piece_id=GCFI63PMDS2MTX6WHFW3MKFPOTXAWVI6HQVWNVRUBEXZ6UVFUQNQ, seg_pos={0 0}, upload=449086]: Long tail cancellation due to optimal threshold Dec 26 23:16:06.439706997 (511h47m6.310790032s) [noise=true , address=45.91.116.7:4011 , piece_id=7W5ARDKVKREZSW7BDLI2NUQAMP2MTVKBVTNORKXRV44N5R3SCS4A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:52480->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:52480->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52480->45.91.116.7:4011: use of closed network connection Dec 26 23:16:06.439722807 (511h47m6.310805842s) [noise=true , address=45.91.116.7:4011 , piece_id=7W5ARDKVKREZSW7BDLI2NUQAMP2MTVKBVTNORKXRV44N5R3SCS4A, seg_pos={0 0}, upload=449086]: Long tail cancellation due to optimal threshold Dec 26 23:16:06.439903611 (511h47m6.310986646s) [noise=true , address=77.90.31.16:4735 , piece_id=EDEVQWV4OXGD2I4I4OQ3OTU43UNPPNZGXC3KHONSLXTRGGGIXFMQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:56926->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 23:16:06.439911226 (511h47m6.310994261s) [noise=true , address=77.90.31.16:4735 , piece_id=EDEVQWV4OXGD2I4I4OQ3OTU43UNPPNZGXC3KHONSLXTRGGGIXFMQ, seg_pos={0 0}, upload=449086]: Long tail cancellation due to optimal threshold Dec 26 23:16:06.439927066 (511h47m6.311010111s) [seg_pos={0 0}, upload=449086]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:16:06.439996107 (511h47m6.311079152s) [upload=449086]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:16:06.440001427 (511h47m6.311084472s) [upload=449086]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:16:06.737326335 (511h47m6.60840938s) [upload=449086]: Done uploading segments... Dec 26 23:16:06.737353497 (511h47m6.608436542s) [upload=449086]: Upload finished. err: <nil> Duration: 6.172247148s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.280162894s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:16:39 UTC) orbiter warning: context deadline exceeded Duration: 6.280187411s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX.....X....X.......X.......X..............................X] duration: 30.00172731s trace: https://jaeger.dev.storj.io/trace/130eb89f4e53f1e0 FAIL (2025-12-26 23:16:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: BJHWKNTZFP7FJCGSDW5PK6QIQIFAWPM2YUHDIK4ULBKOCCRVPFIQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 36QPCOF5BYCWH3FWKFYGTQYNYPITIHJYHEYHNUN74EV3CMUGHR7A) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: SDSXPMCEU34QOVN35VFPVAAHUVCMWDS2SWUMAMCBX3ZZGCFLXEXA) context deadline exceeded ======LOGS====== 449086]: Starting upload Dec 26 23:16:03.827516883 (511h47m3.698599928s) [upload=449086]: Done starting upload Dec 26 23:16:03.827664724 (511h47m3.698747769s) [upload=449086]: Uploading segments... Dec 26 23:16:03.827810051 (511h47m3.698893086s) [upload=449086]: Got next segment. Inline: false Dec 26 23:16:03.827849877 (511h47m3.698932922s) [upload=449086]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:16:03.827854716 (511h47m3.698937751s) [upload=449086]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:16:04.076311273 (511h47m3.947394328s) [seg_pos={0 0}, upload=449086]: Begin upload segment... Dec 26 23:16:04.076387258 (511h47m3.947470303s) [seg_pos={0 0}, upload=449086]: Done begin upload segment. Dec 26 23:16:04.076424929 (511h47m3.947507974s) [noise=true , address=45.91.116.7:4011 , piece_id=7W5ARDKVKREZSW7BDLI2NUQAMP2MTVKBVTNORKXRV44N5R3SCS4A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.077198041 (511h47m3.948281086s) [noise=true , address=77.90.31.16:4735 , piece_id=EDEVQWV4OXGD2I4I4OQ3OTU43UNPPNZGXC3KHONSLXTRGGGIXFMQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.077396269 (511h47m3.948479304s) [noise=true , address=145.79.227.51:4409 , piece_id=NW3YMT4WJOTESBG4VLZMA5QLC6QBSMFQ6RUE7QJHRRQSDHUCTPIQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.077886262 (511h47m3.948969297s) [noise=true , address=45.91.116.5:4298 , piece_id=J4VPBFSGIYAFU3QJRNVK26XOPBVXCZZUHOTPRXITLNDXEFRVLH4A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.078070983 (511h47m3.949154019s) [noise=true , address=145.79.227.50:4373 , piece_id=PYWDDOIZMFWSODZHE2INI3VGFSUOCGSIK4MCTVET2VC563PJVLWA, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.078379481 (511h47m3.949462526s) [noise=true , address=145.79.227.52:4446 , piece_id=HIWZ5KJ5NS6E4OEU5THWMYXRSFGBLR5XRQTJQQKRUR6URZKXDVGQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.078766578 (511h47m3.949849623s) [noise=true , address=45.91.116.11:4119 , piece_id=SDSXPMCEU34QOVN35VFPVAAHUVCMWDS2SWUMAMCBX3ZZGCFLXEXA, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.079195546 (511h47m3.950278581s) [noise=true , address=145.79.227.49:4338 , piece_id=XUC5VF2KVI2M3WSXME6TTPJDTZD5WPWOLOJEQDD67W2QKDDW345A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.079657595 (511h47m3.95074064s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VRI5ZGK2XLDLT6MBPJUXXTF4JGLVBQFAALJUDSRL4YXAJKQ5ZA, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.080066523 (511h47m3.951149558s) [noise=true , address=145.79.227.47:4267 , piece_id=ANZOJFHTGGWU3HE24OHZ7YWF5M6ONZBVQVYFTWRDN6PWMUS2IQNQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.080149692 (511h47m3.951232727s) [noise=true , address=45.91.116.4:4939 , piece_id=2LDMFLULUFKDEM7FRFMJP5STS6KDD6XAFCQJE7BFLCOT26ELAFAA, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.080585060 (511h47m3.951668106s) [noise=true , address=145.79.227.38:4978 , piece_id=3L5EOFRYYHYUGUAXH6LFCB7JLDNF3ASWMVFBCDF7Z7NMD3INHU5A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.080969733 (511h47m3.952052778s) [noise=true , address=45.91.116.12:4228 , piece_id=BJHWKNTZFP7FJCGSDW5PK6QIQIFAWPM2YUHDIK4ULBKOCCRVPFIQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.081353323 (511h47m3.952436368s) [noise=true , address=145.79.227.37:4567 , piece_id=PSOXFGFGPNSFWQRP2C3WOBI6ERTLA4K4AZNKS2WB2DKPAK6DWOPQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.081787120 (511h47m3.952870165s) [noise=true , address=45.91.116.8:4046 , piece_id=36QPCOF5BYCWH3FWKFYGTQYNYPITIHJYHEYHNUN74EV3CMUGHR7A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.081961282 (511h47m3.953044327s) [noise=true , address=45.91.116.9:4263 , piece_id=GCFI63PMDS2MTX6WHFW3MKFPOTXAWVI6HQVWNVRUBEXZ6UVFUQNQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:05.800268217 (511h47m5.671351252s) [upload=449086]: Next returned nil segment Dec 26 23:16:05.800289818 (511h47m5.671372863s) [upload=449086]: Waiting for error group managing segments... Dec 26 23:16:06.086124055 (511h47m5.95720709s) [noise=true , address=145.79.227.50:4373 , piece_id=PYWDDOIZMFWSODZHE2INI3VGFSUOCGSIK4MCTVET2VC563PJVLWA, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.095285099 (511h47m5.966368135s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VRI5ZGK2XLDLT6MBPJUXXTF4JGLVBQFAALJUDSRL4YXAJKQ5ZA, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.096266187 (511h47m5.967349233s) [noise=true , address=145.79.227.51:4409 , piece_id=NW3YMT4WJOTESBG4VLZMA5QLC6QBSMFQ6RUE7QJHRRQSDHUCTPIQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.099479436 (511h47m5.970562471s) [noise=true , address=145.79.227.52:4446 , piece_id=HIWZ5KJ5NS6E4OEU5THWMYXRSFGBLR5XRQTJQQKRUR6URZKXDVGQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.099654379 (511h47m5.970737444s) [noise=true , address=145.79.227.37:4567 , piece_id=PSOXFGFGPNSFWQRP2C3WOBI6ERTLA4K4AZNKS2WB2DKPAK6DWOPQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.100010197 (511h47m5.971093242s) [noise=true , address=145.79.227.49:4338 , piece_id=XUC5VF2KVI2M3WSXME6TTPJDTZD5WPWOLOJEQDD67W2QKDDW345A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.102804658 (511h47m5.973887693s) [noise=true , address=145.79.227.38:4978 , piece_id=3L5EOFRYYHYUGUAXH6LFCB7JLDNF3ASWMVFBCDF7Z7NMD3INHU5A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.114909597 (511h47m5.985992642s) [noise=true , address=145.79.227.47:4267 , piece_id=ANZOJFHTGGWU3HE24OHZ7YWF5M6ONZBVQVYFTWRDN6PWMUS2IQNQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.327932642 (511h47m6.199015677s) [noise=true , address=45.91.116.12:4228 , piece_id=BJHWKNTZFP7FJCGSDW5PK6QIQIFAWPM2YUHDIK4ULBKOCCRVPFIQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.356727242 (511h47m6.227810287s) [noise=true , address=45.91.116.4:4939 , piece_id=2LDMFLULUFKDEM7FRFMJP5STS6KDD6XAFCQJE7BFLCOT26ELAFAA, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.361363350 (511h47m6.232446395s) [noise=true , address=45.91.116.8:4046 , piece_id=36QPCOF5BYCWH3FWKFYGTQYNYPITIHJYHEYHNUN74EV3CMUGHR7A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.388538935 (511h47m6.25962198s) [noise=true , address=45.91.116.5:4298 , piece_id=J4VPBFSGIYAFU3QJRNVK26XOPBVXCZZUHOTPRXITLNDXEFRVLH4A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.439308448 (511h47m6.310391483s) [noise=true , address=45.91.116.11:4119 , piece_id=SDSXPMCEU34QOVN35VFPVAAHUVCMWDS2SWUMAMCBX3ZZGCFLXEXA, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.439329067 (511h47m6.310412102s) [seg_pos={0 0}, upload=449086]: Segment reached optimal threshold of 13 pieces. Dec 26 23:16:06.439539608 (511h47m6.310622643s) [noise=true , address=45.91.116.9:4263 , piece_id=GCFI63PMDS2MTX6WHFW3MKFPOTXAWVI6HQVWNVRUBEXZ6UVFUQNQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 23:16:06.439545950 (511h47m6.310628985s) [noise=true , address=45.91.116.9:4263 , piece_id=GCFI63PMDS2MTX6WHFW3MKFPOTXAWVI6HQVWNVRUBEXZ6UVFUQNQ, seg_pos={0 0}, upload=449086]: Long tail cancellation due to optimal threshold Dec 26 23:16:06.439706997 (511h47m6.310790032s) [noise=true , address=45.91.116.7:4011 , piece_id=7W5ARDKVKREZSW7BDLI2NUQAMP2MTVKBVTNORKXRV44N5R3SCS4A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:52480->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:52480->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52480->45.91.116.7:4011: use of closed network connection Dec 26 23:16:06.439722807 (511h47m6.310805842s) [noise=true , address=45.91.116.7:4011 , piece_id=7W5ARDKVKREZSW7BDLI2NUQAMP2MTVKBVTNORKXRV44N5R3SCS4A, seg_pos={0 0}, upload=449086]: Long tail cancellation due to optimal threshold Dec 26 23:16:06.439903611 (511h47m6.310986646s) [noise=true , address=77.90.31.16:4735 , piece_id=EDEVQWV4OXGD2I4I4OQ3OTU43UNPPNZGXC3KHONSLXTRGGGIXFMQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:56926->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 23:16:06.439911226 (511h47m6.310994261s) [noise=true , address=77.90.31.16:4735 , piece_id=EDEVQWV4OXGD2I4I4OQ3OTU43UNPPNZGXC3KHONSLXTRGGGIXFMQ, seg_pos={0 0}, upload=449086]: Long tail cancellation due to optimal threshold Dec 26 23:16:06.439927066 (511h47m6.311010111s) [seg_pos={0 0}, upload=449086]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:16:06.439996107 (511h47m6.311079152s) [upload=449086]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:16:06.440001427 (511h47m6.311084472s) [upload=449086]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:16:06.737326335 (511h47m6.60840938s) [upload=449086]: Done uploading segments... Dec 26 23:16:06.737353497 (511h47m6.608436542s) [upload=449086]: Upload finished. err: <nil> Duration: 30.001753541s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XX.....X....X.......X.......X..............................] duration: 20.759702242s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:15:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 20.759734734s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.93010071s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:15:29 UTC) orbiter warning: context deadline exceeded Duration: 5.930129767s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XX.....X....X.......X.......X.............................] duration: 13.089817573s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:15:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 13.08984741s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XX.....X....X.......X.......X............................] duration: 26.792985765s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:14:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 26.793014128s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.151395349s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:14:17 UTC) orbiter warning: context deadline exceeded Duration: 6.151420387s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XX.....X....X.......X.......X...........................] duration: 6.078206459s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:13:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.078239443s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.32825283s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:13:04 UTC) orbiter warning: context deadline exceeded Duration: 6.328429065s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.955433517s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:12:45 UTC) orbiter warning: context deadline exceeded Duration: 5.955456672s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XX.....X....X.......X.......X..........................] duration: 6.148614237s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:12:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.148643041s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.451690609s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:12:29 UTC) orbiter warning: context deadline exceeded Duration: 6.451769819s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......XX.....X....X.......X.......X.........................] duration: 6.144146757s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:12:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.144288406s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.948749918s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:11:58 UTC) orbiter warning: context deadline exceeded Duration: 5.948773332s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......XX.....X....X.......X.......X........................] duration: 6.097911301s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:10:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.097939515s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.37286773s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:10:56 UTC) orbiter warning: context deadline exceeded Duration: 6.372895802s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.312954758s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:10:26 UTC) orbiter warning: context deadline exceeded Duration: 6.312981177s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.225810316s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:09:58 UTC) orbiter warning: context deadline exceeded Duration: 6.225823542s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........XX.....X....X.......X.......X.......................] duration: 6.007209195s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:09:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.007235665s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........XX.....X....X.......X.......X......................] duration: 6.07688241s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:09:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.076904613s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.386742704s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:09:05 UTC) orbiter warning: context deadline exceeded Duration: 6.386810213s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........XX.....X....X.......X.......X.....................] duration: 6.072712468s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:08:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.072738977s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.293975417s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:08:00 UTC) orbiter warning: context deadline exceeded Duration: 6.293999843s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........XX.....X....X.......X.......X....................] duration: 6.073707206s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:07:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.073733085s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.352140561s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:07:10 UTC) orbiter warning: context deadline exceeded Duration: 6.352194593s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X...........XX.....X....X.......X.......X...................] duration: 8.361916082s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:06:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 8.36194124s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.37536355s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:06:00 UTC) orbiter warning: context deadline exceeded Duration: 6.375392655s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X...........XX.....X....X.......X.......X..................] duration: 7.026984699s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:05:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 7.027018101s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.32364576s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:04:58 UTC) orbiter warning: context deadline exceeded Duration: 6.3236726s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X...........XX.....X....X.......X.......X.................] duration: 9.916606393s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:04:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 9.91663631s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.901544354s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:04:04 UTC) orbiter warning: context deadline exceeded Duration: 5.901565876s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X...........XX.....X....X.......X.......X................] duration: 6.672693625s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:03:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.672726249s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 14.431090198s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:03:06 UTC) orbiter warning: context deadline exceeded Duration: 14.431112601s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.238164148s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:02:29 UTC) orbiter warning: context deadline exceeded Duration: 6.238188673s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X...........XX.....X....X.......X.......X...............] duration: 15.820382258s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:02:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 15.82040919s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 7.66230451s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:01:52 UTC) orbiter warning: context deadline exceeded Duration: 7.662329997s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X...........XX.....X....X.......X.......X..............] duration: 6.080829194s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:01:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.080853951s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001309726s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:01:05 UTC) orbiter warning: context deadline exceeded Duration: 30.001343891s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.....X...........XX.....X....X.......X.......X.............] duration: 6.160433963s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:00:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.16046451s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX.....X...........XX.....X....X.......X.......X............] duration: 6.060247214s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 22:59:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.06027171s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)