Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X.....X..X...X........................] duration: 46.55336565s trace: https://jaeger.dev.storj.io/trace/37d27399c8bd577a FAIL (2025-12-29 05:37:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 05:18:03.821720823 (565h49m3.692803878s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822185477 (565h49m3.693268522s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822572013 (565h49m3.693655058s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823071774 (565h49m3.69415482s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823476305 (565h49m3.69455936s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823922204 (565h49m3.695005249s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.824347103 (565h49m3.695430148s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.826529919 (565h49m3.697612964s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.828201432 (565h49m3.699284477s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.829987674 (565h49m3.701070719s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.831682460 (565h49m3.702765505s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.833476788 (565h49m3.704559833s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.834480228 (565h49m3.705563274s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835861419 (565h49m3.706944454s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835997778 (565h49m3.707080823s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.837791423 (565h49m3.708874468s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:05.122896476 (565h49m4.993979511s) [upload=496083]: Next returned nil segment Dec 29 05:18:05.122911676 (565h49m4.993994721s) [upload=496083]: Waiting for error group managing segments... Dec 29 05:18:05.476685321 (565h49m5.347768356s) [noise=true , address=64.49.13.26:4946 , piece_id=SLV6JONCSMPVLEW7DM7DVVOLOOSTYE2QEQWXMX2CNLA3VRNQNEZA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.686793637 (565h49m5.557876672s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.695095065 (565h49m5.56617812s) [noise=true , address=145.79.227.3:4162 , piece_id=HCXS3I6KZSTX6EAZSOBDEMVQXX7H3ML6JGNP76Z7XUOALPPCHCOQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.879055562 (565h49m5.750138627s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.952203586 (565h49m5.823286631s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.963348518 (565h49m5.834431563s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.978376062 (565h49m5.849459097s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.029930970 (565h49m5.901014005s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.064081227 (565h49m5.935164272s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.531723043 (565h49m6.402806089s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.534889613 (565h49m6.405972658s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.634538078 (565h49m6.505621123s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.635018352 (565h49m6.506101387s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664932944 (565h49m6.53601599s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664984803 (565h49m6.536067848s) [seg_pos={0 0}, upload=496083]: Segment reached optimal threshold of 14 pieces. Dec 29 05:18:06.665226433 (565h49m6.536309478s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665236311 (565h49m6.536319366s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665410774 (565h49m6.536493819s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GHRhMSQiYGrk84jej1nYkPf1A9dxxP36cPc6suGHuVALqXF4r): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665431724 (565h49m6.536514779s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665544699 (565h49m6.536627744s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection Dec 29 05:18:06.665553366 (565h49m6.536636411s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665695977 (565h49m6.536779033s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665707088 (565h49m6.536790153s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665738198 (565h49m6.536821243s) [seg_pos={0 0}, upload=496083]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:18:06.665904214 (565h49m6.536987259s) [upload=496083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:18:06.665927027 (565h49m6.537010072s) [upload=496083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:18:06.966199067 (565h49m6.837282102s) [upload=496083]: Done uploading segments... Dec 29 05:18:06.966223153 (565h49m6.837306188s) [upload=496083]: Upload finished. err: <nil> Duration: 46.553389095s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [........X..X...X....X....X.........X...X...........X.....X..] duration: 3.504882525s trace: https://jaeger.dev.storj.io/trace/77b1c09c675b64cc FAIL (2025-12-29 05:36:44 UTC) orbiter warning: context deadline exceeded Duration: 3.504910337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X...X..XXX..X............X..X.X...XX...X.XX...........XX] duration: 30.003266276s trace: https://jaeger.dev.storj.io/trace/4b9f8a60f8170153 FAIL (2025-12-29 05:36:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: 4BSA4JMXJTCV3MGJ4HKZJP4IMHONKNZFFQQJIRZZ2RXYOGS2RVQQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: KBXZ57WB6HPZI37R4ITUG6OFVGFITK55GH7VYWEP5SWIQ2G4FLUQ) context deadline exceeded ======LOGS====== Dec 29 05:36:04.560072508 (566h7m4.431155553s) [upload=496354]: Starting upload Dec 29 05:36:04.560135027 (566h7m4.431218072s) [upload=496354]: Done starting upload Dec 29 05:36:04.560291295 (566h7m4.43137433s) [upload=496354]: Uploading segments... Dec 29 05:36:04.560351249 (566h7m4.431434294s) [upload=496354]: Got next segment. Inline: false Dec 29 05:36:04.560381687 (566h7m4.431464722s) [upload=496354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:36:04.560387197 (566h7m4.431470242s) [upload=496354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:36:04.810217578 (566h7m4.681300624s) [seg_pos={0 0}, upload=496354]: Begin upload segment... Dec 29 05:36:04.810288994 (566h7m4.681372039s) [seg_pos={0 0}, upload=496354]: Done begin upload segment. Dec 29 05:36:04.810345552 (566h7m4.681428598s) [noise=true , address=145.79.227.44:4195 , piece_id=EIJDB2PAW2KEM3G7JB5INSGSCIIMURCB7NJKTTKZDNYMSZMDX33Q, seg_pos={0 0}, upload=496354]: Uploading piece... Dec 29 05:36:04.811453082 (566h7m4.682536127s) [noise=true , address=145.79.227.15:4875 , piece_id=XTONML5K7MYXTFK5JKMERB2OHB2RGP2OCFRFXFEJZZHVDOLPTB3Q, seg_pos={0 0}, upload=496354]: Uploading piece... Dec 29 05:36:04.812072691 (566h7m4.683155726s) [noise=true , address=77.90.31.12:4589 , piece_id=3GR76NM6VBBVGFP5IJEPJOWJUVTDJIWMFA5RQLZXTJXIGY2SNT3A, seg_pos={0 0}, upload=496354]: Uploading piece... Dec 29 05:36:04.812547225 (566h7m4.68363027s) [noise=true , address=145.79.227.41:4085 , piece_id=H7S3D52KIXYKL24UMTUQAOM5U5WVBBR4B4L5TY4BH55XDJ2JFBSA, seg_pos={0 0}, upload=496354]: Uploading piece... Dec 29 05:36:04.812958127 (566h7m4.684041172s) [noise=true , address=145.79.227.20:4438 , piece_id=REP7C367OHOAU7HYKITBZKNJHK5X7NHDAG4ZY3B6FN2CAQKQCNPA, seg_pos={0 0}, upload=496354]: Uploading piece... Dec 29 05:36:04.813350624 (566h7m4.684433669s) [noise=true , address=145.79.227.46:4603 , piece_id=BPQUWCNA2WJRWKICPUDL4Z4QKWVUMOQEICNUQQI23QIXNPOJ5TXQ, seg_pos={0 0}, upload=496354]: Uploading piece... Dec 29 05:36:04.813875694 (566h7m4.684958739s) [noise=true , address=145.79.227.26:4653 , piece_id=7QSSJCEPFWWZOHBO63ZAPVJ5UALUOF7ENHFXG3YEEBZREYBDRDBA, seg_pos={0 0}, upload=496354]: Uploading piece... Dec 29 05:36:04.814293971 (566h7m4.685377016s) [noise=true , address=145.79.227.16:4911 , piece_id=AW4VABJGXNCS4FJ44CMYJXI7S74CXN5JKEQXZFLXSNX2YC5UQPWA, seg_pos={0 0}, upload=496354]: Uploading piece... Dec 29 05:36:04.814750790 (566h7m4.685833835s) [noise=true , address=145.79.227.14:4840 , piece_id=6RDD7HOJUZHNSIYSG7EXM767XYPGA7EBCUMC3DXJZHWWGMYWVCQA, seg_pos={0 0}, upload=496354]: Uploading piece... Dec 29 05:36:04.815178706 (566h7m4.686261751s) [noise=true , address=145.79.227.51:4411 , piece_id=2TJPAGZNWI3D7Q4XPEOCTBAHXWK4BYX3VUWIZFKWAUIGB5UGDZIA, seg_pos={0 0}, upload=496354]: Uploading piece... Dec 29 05:36:04.815611139 (566h7m4.686694184s) [noise=true , address=145.79.227.39:4013 , piece_id=VFOLKQ6BYVD5JNTJUCI4D2SEH3NOKRCLDO4RB4FN5PDLVX622MDA, seg_pos={0 0}, upload=496354]: Uploading piece... Dec 29 05:36:04.816014487 (566h7m4.687097532s) [noise=true , address=145.79.227.50:4374 , piece_id=QRSDGJI7UFUKM4OKZVRYNSOFXEAT5J37SXLRXG25VK2A7GNOBN3Q, seg_pos={0 0}, upload=496354]: Uploading piece... Dec 29 05:36:04.816412114 (566h7m4.687495159s) [noise=true , address=77.90.31.10:4517 , piece_id=BNUEEPXQT5AILXUT7PRCQ32MHE6XCRJW4VCI7OO7PYQLEQSFPGIQ, seg_pos={0 0}, upload=496354]: Uploading piece... Dec 29 05:36:04.817019010 (566h7m4.688102055s) [noise=true , address=77.90.31.21:4877 , piece_id=4BSA4JMXJTCV3MGJ4HKZJP4IMHONKNZFFQQJIRZZ2RXYOGS2RVQQ, seg_pos={0 0}, upload=496354]: Uploading piece... Dec 29 05:36:04.817447957 (566h7m4.688531002s) [noise=true , address=45.91.116.6:4973 , piece_id=KBXZ57WB6HPZI37R4ITUG6OFVGFITK55GH7VYWEP5SWIQ2G4FLUQ, seg_pos={0 0}, upload=496354]: Uploading piece... Dec 29 05:36:04.822349160 (566h7m4.693432205s) [noise=true , address=45.91.116.4:4939 , piece_id=Y3GZWZ4UEBVOCVNCIS27C5TUTENL4PQLFCNJBIB6CKLTE3PXHVMQ, seg_pos={0 0}, upload=496354]: Uploading piece... Dec 29 05:36:06.420479642 (566h7m6.291562677s) [upload=496354]: Next returned nil segment Dec 29 05:36:06.420500161 (566h7m6.291583206s) [upload=496354]: Waiting for error group managing segments... Dec 29 05:36:06.806886044 (566h7m6.677969149s) [noise=true , address=145.79.227.46:4603 , piece_id=BPQUWCNA2WJRWKICPUDL4Z4QKWVUMOQEICNUQQI23QIXNPOJ5TXQ, seg_pos={0 0}, upload=496354]: Done uploading piece. err: <nil> Dec 29 05:36:06.814094079 (566h7m6.685177114s) [noise=true , address=145.79.227.41:4085 , piece_id=H7S3D52KIXYKL24UMTUQAOM5U5WVBBR4B4L5TY4BH55XDJ2JFBSA, seg_pos={0 0}, upload=496354]: Done uploading piece. err: <nil> Dec 29 05:36:06.814637995 (566h7m6.68572103s) [noise=true , address=145.79.227.39:4013 , piece_id=VFOLKQ6BYVD5JNTJUCI4D2SEH3NOKRCLDO4RB4FN5PDLVX622MDA, seg_pos={0 0}, upload=496354]: Done uploading piece. err: <nil> Dec 29 05:36:06.815893225 (566h7m6.68697626s) [noise=true , address=145.79.227.14:4840 , piece_id=6RDD7HOJUZHNSIYSG7EXM767XYPGA7EBCUMC3DXJZHWWGMYWVCQA, seg_pos={0 0}, upload=496354]: Done uploading piece. err: <nil> Dec 29 05:36:06.816550005 (566h7m6.687633041s) [noise=true , address=145.79.227.51:4411 , piece_id=2TJPAGZNWI3D7Q4XPEOCTBAHXWK4BYX3VUWIZFKWAUIGB5UGDZIA, seg_pos={0 0}, upload=496354]: Done uploading piece. err: <nil> Dec 29 05:36:06.817034769 (566h7m6.688117804s) [noise=true , address=145.79.227.16:4911 , piece_id=AW4VABJGXNCS4FJ44CMYJXI7S74CXN5JKEQXZFLXSNX2YC5UQPWA, seg_pos={0 0}, upload=496354]: Done uploading piece. err: <nil> Dec 29 05:36:06.819983363 (566h7m6.691066398s) [noise=true , address=145.79.227.50:4374 , piece_id=QRSDGJI7UFUKM4OKZVRYNSOFXEAT5J37SXLRXG25VK2A7GNOBN3Q, seg_pos={0 0}, upload=496354]: Done uploading piece. err: <nil> Dec 29 05:36:06.828052578 (566h7m6.699135613s) [noise=true , address=145.79.227.26:4653 , piece_id=7QSSJCEPFWWZOHBO63ZAPVJ5UALUOF7ENHFXG3YEEBZREYBDRDBA, seg_pos={0 0}, upload=496354]: Done uploading piece. err: <nil> Dec 29 05:36:06.832312740 (566h7m6.703395775s) [noise=true , address=145.79.227.15:4875 , piece_id=XTONML5K7MYXTFK5JKMERB2OHB2RGP2OCFRFXFEJZZHVDOLPTB3Q, seg_pos={0 0}, upload=496354]: Done uploading piece. err: <nil> Dec 29 05:36:06.840479740 (566h7m6.711562775s) [noise=true , address=145.79.227.20:4438 , piece_id=REP7C367OHOAU7HYKITBZKNJHK5X7NHDAG4ZY3B6FN2CAQKQCNPA, seg_pos={0 0}, upload=496354]: Done uploading piece. err: <nil> Dec 29 05:36:07.077969194 (566h7m6.949052249s) [noise=true , address=77.90.31.12:4589 , piece_id=3GR76NM6VBBVGFP5IJEPJOWJUVTDJIWMFA5RQLZXTJXIGY2SNT3A, seg_pos={0 0}, upload=496354]: Done uploading piece. err: <nil> Dec 29 05:36:07.099073680 (566h7m6.970156715s) [noise=true , address=45.91.116.6:4973 , piece_id=KBXZ57WB6HPZI37R4ITUG6OFVGFITK55GH7VYWEP5SWIQ2G4FLUQ, seg_pos={0 0}, upload=496354]: Done uploading piece. err: <nil> Dec 29 05:36:07.675925035 (566h7m7.54700808s) [noise=true , address=77.90.31.21:4877 , piece_id=4BSA4JMXJTCV3MGJ4HKZJP4IMHONKNZFFQQJIRZZ2RXYOGS2RVQQ, seg_pos={0 0}, upload=496354]: Done uploading piece. err: <nil> Dec 29 05:36:07.675957567 (566h7m7.547040612s) [seg_pos={0 0}, upload=496354]: Segment reached optimal threshold of 13 pieces. Dec 29 05:36:07.676273488 (566h7m7.547356523s) [noise=true , address=145.79.227.44:4195 , piece_id=EIJDB2PAW2KEM3G7JB5INSGSCIIMURCB7NJKTTKZDNYMSZMDX33Q, seg_pos={0 0}, upload=496354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 29 05:36:07.676281784 (566h7m7.547364829s) [noise=true , address=145.79.227.44:4195 , piece_id=EIJDB2PAW2KEM3G7JB5INSGSCIIMURCB7NJKTTKZDNYMSZMDX33Q, seg_pos={0 0}, upload=496354]: Long tail cancellation due to optimal threshold Dec 29 05:36:07.676472256 (566h7m7.547555311s) [noise=true , address=77.90.31.10:4517 , piece_id=BNUEEPXQT5AILXUT7PRCQ32MHE6XCRJW4VCI7OO7PYQLEQSFPGIQ, seg_pos={0 0}, upload=496354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 29 05:36:07.676495781 (566h7m7.547578937s) [noise=true , address=77.90.31.10:4517 , piece_id=BNUEEPXQT5AILXUT7PRCQ32MHE6XCRJW4VCI7OO7PYQLEQSFPGIQ, seg_pos={0 0}, upload=496354]: Long tail cancellation due to optimal threshold Dec 29 05:36:07.676552700 (566h7m7.547635745s) [noise=true , address=45.91.116.4:4939 , piece_id=Y3GZWZ4UEBVOCVNCIS27C5TUTENL4PQLFCNJBIB6CKLTE3PXHVMQ, seg_pos={0 0}, upload=496354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39386->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 05:36:07.676562418 (566h7m7.547645463s) [noise=true , address=45.91.116.4:4939 , piece_id=Y3GZWZ4UEBVOCVNCIS27C5TUTENL4PQLFCNJBIB6CKLTE3PXHVMQ, seg_pos={0 0}, upload=496354]: Long tail cancellation due to optimal threshold Dec 29 05:36:07.676583268 (566h7m7.547666303s) [seg_pos={0 0}, upload=496354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:36:07.676725299 (566h7m7.547808334s) [upload=496354]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:36:07.676730749 (566h7m7.547813784s) [upload=496354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:36:07.977782693 (566h7m7.848865728s) [upload=496354]: Done uploading segments... Dec 29 05:36:07.977825744 (566h7m7.848908779s) [upload=496354]: Upload finished. err: <nil> Duration: 30.003287356s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X........X..X...X....X....X.........X...X...........X.....X.] duration: 4.820895863s trace: https://jaeger.dev.storj.io/trace/77b1c09c675b64cc FAIL (2025-12-29 05:36:18 UTC) orbiter warning: context deadline exceeded Duration: 4.82094078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X.....X..X...X.......................] duration: 49.591923293s trace: https://jaeger.dev.storj.io/trace/37d27399c8bd577a FAIL (2025-12-29 05:36:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 05:18:03.821720823 (565h49m3.692803878s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822185477 (565h49m3.693268522s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822572013 (565h49m3.693655058s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823071774 (565h49m3.69415482s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823476305 (565h49m3.69455936s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823922204 (565h49m3.695005249s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.824347103 (565h49m3.695430148s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.826529919 (565h49m3.697612964s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.828201432 (565h49m3.699284477s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.829987674 (565h49m3.701070719s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.831682460 (565h49m3.702765505s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.833476788 (565h49m3.704559833s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.834480228 (565h49m3.705563274s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835861419 (565h49m3.706944454s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835997778 (565h49m3.707080823s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.837791423 (565h49m3.708874468s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:05.122896476 (565h49m4.993979511s) [upload=496083]: Next returned nil segment Dec 29 05:18:05.122911676 (565h49m4.993994721s) [upload=496083]: Waiting for error group managing segments... Dec 29 05:18:05.476685321 (565h49m5.347768356s) [noise=true , address=64.49.13.26:4946 , piece_id=SLV6JONCSMPVLEW7DM7DVVOLOOSTYE2QEQWXMX2CNLA3VRNQNEZA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.686793637 (565h49m5.557876672s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.695095065 (565h49m5.56617812s) [noise=true , address=145.79.227.3:4162 , piece_id=HCXS3I6KZSTX6EAZSOBDEMVQXX7H3ML6JGNP76Z7XUOALPPCHCOQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.879055562 (565h49m5.750138627s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.952203586 (565h49m5.823286631s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.963348518 (565h49m5.834431563s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.978376062 (565h49m5.849459097s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.029930970 (565h49m5.901014005s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.064081227 (565h49m5.935164272s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.531723043 (565h49m6.402806089s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.534889613 (565h49m6.405972658s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.634538078 (565h49m6.505621123s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.635018352 (565h49m6.506101387s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664932944 (565h49m6.53601599s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664984803 (565h49m6.536067848s) [seg_pos={0 0}, upload=496083]: Segment reached optimal threshold of 14 pieces. Dec 29 05:18:06.665226433 (565h49m6.536309478s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665236311 (565h49m6.536319366s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665410774 (565h49m6.536493819s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GHRhMSQiYGrk84jej1nYkPf1A9dxxP36cPc6suGHuVALqXF4r): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665431724 (565h49m6.536514779s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665544699 (565h49m6.536627744s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection Dec 29 05:18:06.665553366 (565h49m6.536636411s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665695977 (565h49m6.536779033s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665707088 (565h49m6.536790153s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665738198 (565h49m6.536821243s) [seg_pos={0 0}, upload=496083]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:18:06.665904214 (565h49m6.536987259s) [upload=496083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:18:06.665927027 (565h49m6.537010072s) [upload=496083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:18:06.966199067 (565h49m6.837282102s) [upload=496083]: Done uploading segments... Dec 29 05:18:06.966223153 (565h49m6.837306188s) [upload=496083]: Upload finished. err: <nil> Duration: 49.591950045s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X...X..XXX..X............X..X.X...XX...X.XX...........X] duration: 30.0041561s trace: https://jaeger.dev.storj.io/trace/2adcba20d3d0b8cd FAIL (2025-12-29 05:35:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: DAHQCVSBHFHXJ7HIZ7ZI5OD32DSNFS4JKVXSXM27G73JC56ONBSA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 5QGOBJUVNSMQ5VPK7NOTM73HPYYIMDOBDKHJWXCGTMHR2HQBGA4Q) context deadline exceeded ======LOGS====== Dec 29 05:35:00.541474904 (566h6m0.412557949s) [upload=496341]: Starting upload Dec 29 05:35:00.541550448 (566h6m0.412633483s) [upload=496341]: Done starting upload Dec 29 05:35:00.541666819 (566h6m0.412749865s) [upload=496341]: Uploading segments... Dec 29 05:35:00.542041002 (566h6m0.413124057s) [upload=496341]: Got next segment. Inline: false Dec 29 05:35:00.542170909 (566h6m0.413253964s) [upload=496341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:35:00.542201096 (566h6m0.413284142s) [upload=496341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:35:00.794258520 (566h6m0.665341565s) [seg_pos={0 0}, upload=496341]: Begin upload segment... Dec 29 05:35:00.794302072 (566h6m0.665385117s) [seg_pos={0 0}, upload=496341]: Done begin upload segment. Dec 29 05:35:00.794341547 (566h6m0.665424582s) [noise=true , address=145.79.227.36:4518 , piece_id=KVSG7A2BFPM64H4Y3GDT5JPAESLDUEDBJXGBQBV7H3WFOOZHMRIA, seg_pos={0 0}, upload=496341]: Uploading piece... Dec 29 05:35:00.795287558 (566h6m0.666370593s) [noise=true , address=145.79.227.2:4114 , piece_id=7KLHM5VMCQFGUV63DKEAS5K3FYS36K7LXRQEGAXWX2536VUEIRZQ, seg_pos={0 0}, upload=496341]: Uploading piece... Dec 29 05:35:00.795431743 (566h6m0.666514808s) [noise=true , address=145.79.227.26:4655 , piece_id=B2JDLD5P24QTOJN52CXLSLTU7N3GDGQJVCZELBBG5G77XVLEK6YA, seg_pos={0 0}, upload=496341]: Uploading piece... Dec 29 05:35:00.795594423 (566h6m0.666677468s) [noise=true , address=145.79.227.48:4301 , piece_id=K5KT4K5AGRIAJXBHEBVOOYUQ53EJR3ZLGOSZ2X7CE7YG6KJWTRXQ, seg_pos={0 0}, upload=496341]: Uploading piece... Dec 29 05:35:00.795785166 (566h6m0.666868211s) [noise=true , address=145.79.227.13:4804 , piece_id=XA5R3TMKK5K67IFENPFSO6DZHNT5XIDRFKR4EVE66PL5HUA5WDQA, seg_pos={0 0}, upload=496341]: Uploading piece... Dec 29 05:35:00.795931044 (566h6m0.667014079s) [noise=true , address=45.91.116.14:4191 , piece_id=QV7KK6HS4XLMHPI2M5HDGRMZ4NG7M3LS2LPMZBU26HI4FSB4ZUFA, seg_pos={0 0}, upload=496341]: Uploading piece... Dec 29 05:35:00.796158607 (566h6m0.667241672s) [noise=true , address=77.90.31.17:4770 , piece_id=SDRR2AM4NBKGWC5QZXVJFMOY7IKFQ3YDLFKFBCVYMLCXUHJONN7A, seg_pos={0 0}, upload=496341]: Uploading piece... Dec 29 05:35:00.796327248 (566h6m0.667410293s) [noise=true , address=145.79.227.12:4767 , piece_id=WX4Y2UAHRGZR325LAN7M2IDAMQF3CQ3T5HXUO34WSIEZ2KSCZIOQ, seg_pos={0 0}, upload=496341]: Uploading piece... Dec 29 05:35:00.796498194 (566h6m0.667581229s) [noise=true , address=145.79.227.4:4187 , piece_id=IXGBDD2QHXNAZYIXOV4MKBDN73DC3GJ5WFZXA4ZU5XOWRMOMJRGQ, seg_pos={0 0}, upload=496341]: Uploading piece... Dec 29 05:35:00.796814867 (566h6m0.667897902s) [noise=true , address=145.79.227.39:4015 , piece_id=A5BA7NGQY3F72K7FVTS4V2G6PWIPBSOK56V62HTQOXAB7V4D35LQ, seg_pos={0 0}, upload=496341]: Uploading piece... Dec 29 05:35:00.796890731 (566h6m0.667973776s) [noise=true , address=77.90.31.19:4842 , piece_id=DAHQCVSBHFHXJ7HIZ7ZI5OD32DSNFS4JKVXSXM27G73JC56ONBSA, seg_pos={0 0}, upload=496341]: Uploading piece... Dec 29 05:35:00.797408086 (566h6m0.668491131s) [noise=true , address=145.79.227.34:4906 , piece_id=BO467MBJ2GOKHPBMF5GLGJOHMKHPRLGOIWIQAELBIDAYKDYEWGKQ, seg_pos={0 0}, upload=496341]: Uploading piece... Dec 29 05:35:00.797710162 (566h6m0.668793197s) [noise=true , address=145.79.227.11:4732 , piece_id=SLBEIEXLOIF6QNAUI7KDDJKIBWJ3LGDG6TOWSVVN64VACHDBTE6A, seg_pos={0 0}, upload=496341]: Uploading piece... Dec 29 05:35:00.798033648 (566h6m0.669116693s) [noise=true , address=145.79.227.20:4438 , piece_id=2SQT55VS7SHI5ILF57SCXUD4UBFM6AZHWGQL2IANXKNPWR5VXSPA, seg_pos={0 0}, upload=496341]: Uploading piece... Dec 29 05:35:00.798368955 (566h6m0.669452001s) [noise=true , address=45.91.116.10:4084 , piece_id=5QGOBJUVNSMQ5VPK7NOTM73HPYYIMDOBDKHJWXCGTMHR2HQBGA4Q, seg_pos={0 0}, upload=496341]: Uploading piece... Dec 29 05:35:00.798522108 (566h6m0.669605143s) [noise=true , address=145.79.227.38:4979 , piece_id=G52AUWVBCFWA3B25NAD6ZULXQTFIOKNYYHWGOMDUNV2TPKU3KOQA, seg_pos={0 0}, upload=496341]: Uploading piece... Dec 29 05:35:02.436700268 (566h6m2.307783313s) [upload=496341]: Next returned nil segment Dec 29 05:35:02.436718173 (566h6m2.307801228s) [upload=496341]: Waiting for error group managing segments... Dec 29 05:35:02.802843256 (566h6m2.673926311s) [noise=true , address=145.79.227.20:4438 , piece_id=2SQT55VS7SHI5ILF57SCXUD4UBFM6AZHWGQL2IANXKNPWR5VXSPA, seg_pos={0 0}, upload=496341]: Done uploading piece. err: <nil> Dec 29 05:35:02.807459546 (566h6m2.678542591s) [noise=true , address=145.79.227.2:4114 , piece_id=7KLHM5VMCQFGUV63DKEAS5K3FYS36K7LXRQEGAXWX2536VUEIRZQ, seg_pos={0 0}, upload=496341]: Done uploading piece. err: <nil> Dec 29 05:35:02.810351623 (566h6m2.681434668s) [noise=true , address=145.79.227.36:4518 , piece_id=KVSG7A2BFPM64H4Y3GDT5JPAESLDUEDBJXGBQBV7H3WFOOZHMRIA, seg_pos={0 0}, upload=496341]: Done uploading piece. err: <nil> Dec 29 05:35:02.813756306 (566h6m2.684839362s) [noise=true , address=145.79.227.12:4767 , piece_id=WX4Y2UAHRGZR325LAN7M2IDAMQF3CQ3T5HXUO34WSIEZ2KSCZIOQ, seg_pos={0 0}, upload=496341]: Done uploading piece. err: <nil> Dec 29 05:35:02.813927823 (566h6m2.685010888s) [noise=true , address=145.79.227.26:4655 , piece_id=B2JDLD5P24QTOJN52CXLSLTU7N3GDGQJVCZELBBG5G77XVLEK6YA, seg_pos={0 0}, upload=496341]: Done uploading piece. err: <nil> Dec 29 05:35:02.814772191 (566h6m2.685855236s) [noise=true , address=145.79.227.48:4301 , piece_id=K5KT4K5AGRIAJXBHEBVOOYUQ53EJR3ZLGOSZ2X7CE7YG6KJWTRXQ, seg_pos={0 0}, upload=496341]: Done uploading piece. err: <nil> Dec 29 05:35:02.817148014 (566h6m2.688231069s) [noise=true , address=145.79.227.4:4187 , piece_id=IXGBDD2QHXNAZYIXOV4MKBDN73DC3GJ5WFZXA4ZU5XOWRMOMJRGQ, seg_pos={0 0}, upload=496341]: Done uploading piece. err: <nil> Dec 29 05:35:02.823017891 (566h6m2.694100936s) [noise=true , address=145.79.227.13:4804 , piece_id=XA5R3TMKK5K67IFENPFSO6DZHNT5XIDRFKR4EVE66PL5HUA5WDQA, seg_pos={0 0}, upload=496341]: Done uploading piece. err: <nil> Dec 29 05:35:02.824736604 (566h6m2.695819649s) [noise=true , address=145.79.227.38:4979 , piece_id=G52AUWVBCFWA3B25NAD6ZULXQTFIOKNYYHWGOMDUNV2TPKU3KOQA, seg_pos={0 0}, upload=496341]: Done uploading piece. err: <nil> Dec 29 05:35:02.841007375 (566h6m2.71209042s) [noise=true , address=145.79.227.11:4732 , piece_id=SLBEIEXLOIF6QNAUI7KDDJKIBWJ3LGDG6TOWSVVN64VACHDBTE6A, seg_pos={0 0}, upload=496341]: Done uploading piece. err: <nil> Dec 29 05:35:02.849404666 (566h6m2.720487711s) [noise=true , address=145.79.227.39:4015 , piece_id=A5BA7NGQY3F72K7FVTS4V2G6PWIPBSOK56V62HTQOXAB7V4D35LQ, seg_pos={0 0}, upload=496341]: Done uploading piece. err: <nil> Dec 29 05:35:03.077129606 (566h6m2.948212651s) [noise=true , address=45.91.116.10:4084 , piece_id=5QGOBJUVNSMQ5VPK7NOTM73HPYYIMDOBDKHJWXCGTMHR2HQBGA4Q, seg_pos={0 0}, upload=496341]: Done uploading piece. err: <nil> Dec 29 05:35:03.204022197 (566h6m3.075105252s) [noise=true , address=77.90.31.19:4842 , piece_id=DAHQCVSBHFHXJ7HIZ7ZI5OD32DSNFS4JKVXSXM27G73JC56ONBSA, seg_pos={0 0}, upload=496341]: Done uploading piece. err: <nil> Dec 29 05:35:03.204051583 (566h6m3.075134638s) [seg_pos={0 0}, upload=496341]: Segment reached optimal threshold of 13 pieces. Dec 29 05:35:03.204275980 (566h6m3.075359025s) [noise=true , address=77.90.31.17:4770 , piece_id=SDRR2AM4NBKGWC5QZXVJFMOY7IKFQ3YDLFKFBCVYMLCXUHJONN7A, seg_pos={0 0}, upload=496341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 29 05:35:03.204284265 (566h6m3.075367311s) [noise=true , address=77.90.31.17:4770 , piece_id=SDRR2AM4NBKGWC5QZXVJFMOY7IKFQ3YDLFKFBCVYMLCXUHJONN7A, seg_pos={0 0}, upload=496341]: Long tail cancellation due to optimal threshold Dec 29 05:35:03.204390768 (566h6m3.075473823s) [noise=true , address=45.91.116.14:4191 , piece_id=QV7KK6HS4XLMHPI2M5HDGRMZ4NG7M3LS2LPMZBU26HI4FSB4ZUFA, seg_pos={0 0}, upload=496341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 05:35:03.204410065 (566h6m3.075493111s) [noise=true , address=145.79.227.34:4906 , piece_id=BO467MBJ2GOKHPBMF5GLGJOHMKHPRLGOIWIQAELBIDAYKDYEWGKQ, seg_pos={0 0}, upload=496341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 29 05:35:03.204418672 (566h6m3.075501717s) [noise=true , address=145.79.227.34:4906 , piece_id=BO467MBJ2GOKHPBMF5GLGJOHMKHPRLGOIWIQAELBIDAYKDYEWGKQ, seg_pos={0 0}, upload=496341]: Long tail cancellation due to optimal threshold Dec 29 05:35:03.204453869 (566h6m3.075536914s) [noise=true , address=45.91.116.14:4191 , piece_id=QV7KK6HS4XLMHPI2M5HDGRMZ4NG7M3LS2LPMZBU26HI4FSB4ZUFA, seg_pos={0 0}, upload=496341]: Long tail cancellation due to optimal threshold Dec 29 05:35:03.204473807 (566h6m3.075556852s) [seg_pos={0 0}, upload=496341]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:35:03.204609736 (566h6m3.075692771s) [upload=496341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:35:03.204616759 (566h6m3.075699804s) [upload=496341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:35:03.518221445 (566h6m3.38930449s) [upload=496341]: Done uploading segments... Dec 29 05:35:03.518248976 (566h6m3.389332021s) [upload=496341]: Upload finished. err: <nil> Duration: 30.004173242s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.X........X..X...X....X....X.........X...X...........X.....X] duration: 5.00106872s trace: https://jaeger.dev.storj.io/trace/77b1c09c675b64cc FAIL (2025-12-29 05:35:19 UTC) orbiter warning: context deadline exceeded Duration: 5.001108125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [......X...X..XXX..X............X..X.X...XX...X.XX...........] duration: 5.916618878s trace: https://jaeger.dev.storj.io/trace/51fc923bdb532cd3 FAIL (2025-12-29 05:34:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ) context deadline exceeded ======LOGS====== c 29 05:28:34.488071392 (565h59m34.359154447s) [upload=496233]: Starting upload Dec 29 05:28:34.488166162 (565h59m34.359249217s) [upload=496233]: Done starting upload Dec 29 05:28:34.488218572 (565h59m34.359301607s) [upload=496233]: Uploading segments... Dec 29 05:28:34.488685831 (565h59m34.359768876s) [upload=496233]: Got next segment. Inline: false Dec 29 05:28:34.488733423 (565h59m34.359816458s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:28:34.488738642 (565h59m34.359821698s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:28:34.740915392 (565h59m34.611998427s) [seg_pos={0 0}, upload=496233]: Begin upload segment... Dec 29 05:28:34.740964546 (565h59m34.612047582s) [seg_pos={0 0}, upload=496233]: Done begin upload segment. Dec 29 05:28:34.740995735 (565h59m34.61207877s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741051191 (565h59m34.612134236s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741822910 (565h59m34.612905945s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742270974 (565h59m34.613354009s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742535387 (565h59m34.613618433s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742691254 (565h59m34.613774299s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743128357 (565h59m34.614211392s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743286116 (565h59m34.614369162s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743574767 (565h59m34.614657802s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743844680 (565h59m34.614927725s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743982262 (565h59m34.615065307s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744390209 (565h59m34.615473254s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744630286 (565h59m34.615713331s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744792054 (565h59m34.615875089s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745257210 (565h59m34.616340245s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745980838 (565h59m34.617063873s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:36.432947041 (565h59m36.304030076s) [upload=496233]: Next returned nil segment Dec 29 05:28:36.432985485 (565h59m36.30406854s) [upload=496233]: Waiting for error group managing segments... Dec 29 05:28:36.725517678 (565h59m36.596600713s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.728084786 (565h59m36.599167821s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.731815750 (565h59m36.602898785s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.733126446 (565h59m36.604209481s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.738813094 (565h59m36.609896129s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.744584674 (565h59m36.615667719s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.747350711 (565h59m36.618433746s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.011810150 (565h59m36.882893195s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.080270838 (565h59m36.951353883s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.087131732 (565h59m36.958214777s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.089270565 (565h59m36.9603536s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.145593362 (565h59m37.016676397s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926865657 (565h59m37.797948692s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926885936 (565h59m37.797968981s) [seg_pos={0 0}, upload=496233]: Segment reached optimal threshold of 13 pieces. Dec 29 05:28:37.927045941 (565h59m37.798128986s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection Dec 29 05:28:37.927051942 (565h59m37.798134987s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927068523 (565h59m37.798151558s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927076979 (565h59m37.798160024s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927265529 (565h59m37.798348574s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927273574 (565h59m37.798356609s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927290146 (565h59m37.798373191s) [seg_pos={0 0}, upload=496233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:28:37.927354828 (565h59m37.798437873s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:28:37.927368705 (565h59m37.79845174s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:28:38.221000271 (565h59m38.092083316s) [upload=496233]: Done uploading segments... Dec 29 05:28:38.221019188 (565h59m38.092102223s) [upload=496233]: Upload finished. err: <nil> Duration: 5.916644878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................X.....X..X...X......................] duration: 22.796427865s trace: https://jaeger.dev.storj.io/trace/37d27399c8bd577a FAIL (2025-12-29 05:34:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 05:18:03.821720823 (565h49m3.692803878s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822185477 (565h49m3.693268522s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822572013 (565h49m3.693655058s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823071774 (565h49m3.69415482s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823476305 (565h49m3.69455936s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823922204 (565h49m3.695005249s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.824347103 (565h49m3.695430148s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.826529919 (565h49m3.697612964s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.828201432 (565h49m3.699284477s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.829987674 (565h49m3.701070719s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.831682460 (565h49m3.702765505s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.833476788 (565h49m3.704559833s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.834480228 (565h49m3.705563274s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835861419 (565h49m3.706944454s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835997778 (565h49m3.707080823s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.837791423 (565h49m3.708874468s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:05.122896476 (565h49m4.993979511s) [upload=496083]: Next returned nil segment Dec 29 05:18:05.122911676 (565h49m4.993994721s) [upload=496083]: Waiting for error group managing segments... Dec 29 05:18:05.476685321 (565h49m5.347768356s) [noise=true , address=64.49.13.26:4946 , piece_id=SLV6JONCSMPVLEW7DM7DVVOLOOSTYE2QEQWXMX2CNLA3VRNQNEZA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.686793637 (565h49m5.557876672s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.695095065 (565h49m5.56617812s) [noise=true , address=145.79.227.3:4162 , piece_id=HCXS3I6KZSTX6EAZSOBDEMVQXX7H3ML6JGNP76Z7XUOALPPCHCOQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.879055562 (565h49m5.750138627s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.952203586 (565h49m5.823286631s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.963348518 (565h49m5.834431563s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.978376062 (565h49m5.849459097s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.029930970 (565h49m5.901014005s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.064081227 (565h49m5.935164272s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.531723043 (565h49m6.402806089s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.534889613 (565h49m6.405972658s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.634538078 (565h49m6.505621123s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.635018352 (565h49m6.506101387s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664932944 (565h49m6.53601599s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664984803 (565h49m6.536067848s) [seg_pos={0 0}, upload=496083]: Segment reached optimal threshold of 14 pieces. Dec 29 05:18:06.665226433 (565h49m6.536309478s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665236311 (565h49m6.536319366s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665410774 (565h49m6.536493819s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GHRhMSQiYGrk84jej1nYkPf1A9dxxP36cPc6suGHuVALqXF4r): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665431724 (565h49m6.536514779s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665544699 (565h49m6.536627744s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection Dec 29 05:18:06.665553366 (565h49m6.536636411s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665695977 (565h49m6.536779033s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665707088 (565h49m6.536790153s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665738198 (565h49m6.536821243s) [seg_pos={0 0}, upload=496083]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:18:06.665904214 (565h49m6.536987259s) [upload=496083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:18:06.665927027 (565h49m6.537010072s) [upload=496083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:18:06.966199067 (565h49m6.837282102s) [upload=496083]: Done uploading segments... Dec 29 05:18:06.966223153 (565h49m6.837306188s) [upload=496083]: Upload finished. err: <nil> Duration: 22.796465427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X......X...X..XXX..X............X..X.X...XX...X.XX..........] duration: 5.644145881s trace: https://jaeger.dev.storj.io/trace/51fc923bdb532cd3 FAIL (2025-12-29 05:34:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ) context deadline exceeded ======LOGS====== c 29 05:28:34.488071392 (565h59m34.359154447s) [upload=496233]: Starting upload Dec 29 05:28:34.488166162 (565h59m34.359249217s) [upload=496233]: Done starting upload Dec 29 05:28:34.488218572 (565h59m34.359301607s) [upload=496233]: Uploading segments... Dec 29 05:28:34.488685831 (565h59m34.359768876s) [upload=496233]: Got next segment. Inline: false Dec 29 05:28:34.488733423 (565h59m34.359816458s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:28:34.488738642 (565h59m34.359821698s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:28:34.740915392 (565h59m34.611998427s) [seg_pos={0 0}, upload=496233]: Begin upload segment... Dec 29 05:28:34.740964546 (565h59m34.612047582s) [seg_pos={0 0}, upload=496233]: Done begin upload segment. Dec 29 05:28:34.740995735 (565h59m34.61207877s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741051191 (565h59m34.612134236s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741822910 (565h59m34.612905945s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742270974 (565h59m34.613354009s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742535387 (565h59m34.613618433s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742691254 (565h59m34.613774299s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743128357 (565h59m34.614211392s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743286116 (565h59m34.614369162s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743574767 (565h59m34.614657802s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743844680 (565h59m34.614927725s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743982262 (565h59m34.615065307s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744390209 (565h59m34.615473254s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744630286 (565h59m34.615713331s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744792054 (565h59m34.615875089s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745257210 (565h59m34.616340245s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745980838 (565h59m34.617063873s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:36.432947041 (565h59m36.304030076s) [upload=496233]: Next returned nil segment Dec 29 05:28:36.432985485 (565h59m36.30406854s) [upload=496233]: Waiting for error group managing segments... Dec 29 05:28:36.725517678 (565h59m36.596600713s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.728084786 (565h59m36.599167821s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.731815750 (565h59m36.602898785s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.733126446 (565h59m36.604209481s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.738813094 (565h59m36.609896129s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.744584674 (565h59m36.615667719s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.747350711 (565h59m36.618433746s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.011810150 (565h59m36.882893195s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.080270838 (565h59m36.951353883s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.087131732 (565h59m36.958214777s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.089270565 (565h59m36.9603536s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.145593362 (565h59m37.016676397s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926865657 (565h59m37.797948692s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926885936 (565h59m37.797968981s) [seg_pos={0 0}, upload=496233]: Segment reached optimal threshold of 13 pieces. Dec 29 05:28:37.927045941 (565h59m37.798128986s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection Dec 29 05:28:37.927051942 (565h59m37.798134987s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927068523 (565h59m37.798151558s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927076979 (565h59m37.798160024s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927265529 (565h59m37.798348574s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927273574 (565h59m37.798356609s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927290146 (565h59m37.798373191s) [seg_pos={0 0}, upload=496233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:28:37.927354828 (565h59m37.798437873s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:28:37.927368705 (565h59m37.79845174s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:28:38.221000271 (565h59m38.092083316s) [upload=496233]: Done uploading segments... Dec 29 05:28:38.221019188 (565h59m38.092102223s) [upload=496233]: Upload finished. err: <nil> Duration: 5.64417188s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..X........X..X...X....X....X.........X...X...........X.....] duration: 4.970647226s trace: https://jaeger.dev.storj.io/trace/28112f54d7c3878c FAIL (2025-12-29 05:34:18 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.970675659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................X.....X..X...X.....................] duration: 55.529662298s trace: https://jaeger.dev.storj.io/trace/37d27399c8bd577a FAIL (2025-12-29 05:34:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 05:18:03.821720823 (565h49m3.692803878s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822185477 (565h49m3.693268522s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822572013 (565h49m3.693655058s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823071774 (565h49m3.69415482s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823476305 (565h49m3.69455936s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823922204 (565h49m3.695005249s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.824347103 (565h49m3.695430148s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.826529919 (565h49m3.697612964s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.828201432 (565h49m3.699284477s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.829987674 (565h49m3.701070719s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.831682460 (565h49m3.702765505s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.833476788 (565h49m3.704559833s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.834480228 (565h49m3.705563274s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835861419 (565h49m3.706944454s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835997778 (565h49m3.707080823s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.837791423 (565h49m3.708874468s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:05.122896476 (565h49m4.993979511s) [upload=496083]: Next returned nil segment Dec 29 05:18:05.122911676 (565h49m4.993994721s) [upload=496083]: Waiting for error group managing segments... Dec 29 05:18:05.476685321 (565h49m5.347768356s) [noise=true , address=64.49.13.26:4946 , piece_id=SLV6JONCSMPVLEW7DM7DVVOLOOSTYE2QEQWXMX2CNLA3VRNQNEZA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.686793637 (565h49m5.557876672s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.695095065 (565h49m5.56617812s) [noise=true , address=145.79.227.3:4162 , piece_id=HCXS3I6KZSTX6EAZSOBDEMVQXX7H3ML6JGNP76Z7XUOALPPCHCOQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.879055562 (565h49m5.750138627s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.952203586 (565h49m5.823286631s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.963348518 (565h49m5.834431563s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.978376062 (565h49m5.849459097s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.029930970 (565h49m5.901014005s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.064081227 (565h49m5.935164272s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.531723043 (565h49m6.402806089s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.534889613 (565h49m6.405972658s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.634538078 (565h49m6.505621123s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.635018352 (565h49m6.506101387s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664932944 (565h49m6.53601599s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664984803 (565h49m6.536067848s) [seg_pos={0 0}, upload=496083]: Segment reached optimal threshold of 14 pieces. Dec 29 05:18:06.665226433 (565h49m6.536309478s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665236311 (565h49m6.536319366s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665410774 (565h49m6.536493819s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GHRhMSQiYGrk84jej1nYkPf1A9dxxP36cPc6suGHuVALqXF4r): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665431724 (565h49m6.536514779s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665544699 (565h49m6.536627744s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection Dec 29 05:18:06.665553366 (565h49m6.536636411s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665695977 (565h49m6.536779033s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665707088 (565h49m6.536790153s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665738198 (565h49m6.536821243s) [seg_pos={0 0}, upload=496083]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:18:06.665904214 (565h49m6.536987259s) [upload=496083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:18:06.665927027 (565h49m6.537010072s) [upload=496083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:18:06.966199067 (565h49m6.837282102s) [upload=496083]: Done uploading segments... Dec 29 05:18:06.966223153 (565h49m6.837306188s) [upload=496083]: Upload finished. err: <nil> Duration: 55.52969016s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX......X...X..XXX..X............X..X.X...XX...X.XX.........] duration: 5.987500877s trace: https://jaeger.dev.storj.io/trace/51fc923bdb532cd3 FAIL (2025-12-29 05:33:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ) context deadline exceeded ======LOGS====== c 29 05:28:34.488071392 (565h59m34.359154447s) [upload=496233]: Starting upload Dec 29 05:28:34.488166162 (565h59m34.359249217s) [upload=496233]: Done starting upload Dec 29 05:28:34.488218572 (565h59m34.359301607s) [upload=496233]: Uploading segments... Dec 29 05:28:34.488685831 (565h59m34.359768876s) [upload=496233]: Got next segment. Inline: false Dec 29 05:28:34.488733423 (565h59m34.359816458s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:28:34.488738642 (565h59m34.359821698s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:28:34.740915392 (565h59m34.611998427s) [seg_pos={0 0}, upload=496233]: Begin upload segment... Dec 29 05:28:34.740964546 (565h59m34.612047582s) [seg_pos={0 0}, upload=496233]: Done begin upload segment. Dec 29 05:28:34.740995735 (565h59m34.61207877s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741051191 (565h59m34.612134236s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741822910 (565h59m34.612905945s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742270974 (565h59m34.613354009s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742535387 (565h59m34.613618433s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742691254 (565h59m34.613774299s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743128357 (565h59m34.614211392s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743286116 (565h59m34.614369162s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743574767 (565h59m34.614657802s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743844680 (565h59m34.614927725s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743982262 (565h59m34.615065307s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744390209 (565h59m34.615473254s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744630286 (565h59m34.615713331s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744792054 (565h59m34.615875089s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745257210 (565h59m34.616340245s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745980838 (565h59m34.617063873s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:36.432947041 (565h59m36.304030076s) [upload=496233]: Next returned nil segment Dec 29 05:28:36.432985485 (565h59m36.30406854s) [upload=496233]: Waiting for error group managing segments... Dec 29 05:28:36.725517678 (565h59m36.596600713s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.728084786 (565h59m36.599167821s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.731815750 (565h59m36.602898785s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.733126446 (565h59m36.604209481s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.738813094 (565h59m36.609896129s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.744584674 (565h59m36.615667719s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.747350711 (565h59m36.618433746s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.011810150 (565h59m36.882893195s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.080270838 (565h59m36.951353883s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.087131732 (565h59m36.958214777s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.089270565 (565h59m36.9603536s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.145593362 (565h59m37.016676397s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926865657 (565h59m37.797948692s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926885936 (565h59m37.797968981s) [seg_pos={0 0}, upload=496233]: Segment reached optimal threshold of 13 pieces. Dec 29 05:28:37.927045941 (565h59m37.798128986s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection Dec 29 05:28:37.927051942 (565h59m37.798134987s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927068523 (565h59m37.798151558s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927076979 (565h59m37.798160024s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927265529 (565h59m37.798348574s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927273574 (565h59m37.798356609s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927290146 (565h59m37.798373191s) [seg_pos={0 0}, upload=496233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:28:37.927354828 (565h59m37.798437873s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:28:37.927368705 (565h59m37.79845174s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:28:38.221000271 (565h59m38.092083316s) [upload=496233]: Done uploading segments... Dec 29 05:28:38.221019188 (565h59m38.092102223s) [upload=496233]: Upload finished. err: <nil> Duration: 5.987543319s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XX......X...X..XXX..X............X..X.X...XX...X.XX........] duration: 6.028296993s trace: https://jaeger.dev.storj.io/trace/51fc923bdb532cd3 FAIL (2025-12-29 05:33:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ) context deadline exceeded ======LOGS====== c 29 05:28:34.488071392 (565h59m34.359154447s) [upload=496233]: Starting upload Dec 29 05:28:34.488166162 (565h59m34.359249217s) [upload=496233]: Done starting upload Dec 29 05:28:34.488218572 (565h59m34.359301607s) [upload=496233]: Uploading segments... Dec 29 05:28:34.488685831 (565h59m34.359768876s) [upload=496233]: Got next segment. Inline: false Dec 29 05:28:34.488733423 (565h59m34.359816458s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:28:34.488738642 (565h59m34.359821698s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:28:34.740915392 (565h59m34.611998427s) [seg_pos={0 0}, upload=496233]: Begin upload segment... Dec 29 05:28:34.740964546 (565h59m34.612047582s) [seg_pos={0 0}, upload=496233]: Done begin upload segment. Dec 29 05:28:34.740995735 (565h59m34.61207877s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741051191 (565h59m34.612134236s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741822910 (565h59m34.612905945s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742270974 (565h59m34.613354009s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742535387 (565h59m34.613618433s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742691254 (565h59m34.613774299s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743128357 (565h59m34.614211392s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743286116 (565h59m34.614369162s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743574767 (565h59m34.614657802s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743844680 (565h59m34.614927725s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743982262 (565h59m34.615065307s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744390209 (565h59m34.615473254s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744630286 (565h59m34.615713331s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744792054 (565h59m34.615875089s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745257210 (565h59m34.616340245s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745980838 (565h59m34.617063873s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:36.432947041 (565h59m36.304030076s) [upload=496233]: Next returned nil segment Dec 29 05:28:36.432985485 (565h59m36.30406854s) [upload=496233]: Waiting for error group managing segments... Dec 29 05:28:36.725517678 (565h59m36.596600713s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.728084786 (565h59m36.599167821s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.731815750 (565h59m36.602898785s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.733126446 (565h59m36.604209481s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.738813094 (565h59m36.609896129s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.744584674 (565h59m36.615667719s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.747350711 (565h59m36.618433746s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.011810150 (565h59m36.882893195s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.080270838 (565h59m36.951353883s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.087131732 (565h59m36.958214777s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.089270565 (565h59m36.9603536s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.145593362 (565h59m37.016676397s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926865657 (565h59m37.797948692s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926885936 (565h59m37.797968981s) [seg_pos={0 0}, upload=496233]: Segment reached optimal threshold of 13 pieces. Dec 29 05:28:37.927045941 (565h59m37.798128986s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection Dec 29 05:28:37.927051942 (565h59m37.798134987s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927068523 (565h59m37.798151558s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927076979 (565h59m37.798160024s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927265529 (565h59m37.798348574s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927273574 (565h59m37.798356609s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927290146 (565h59m37.798373191s) [seg_pos={0 0}, upload=496233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:28:37.927354828 (565h59m37.798437873s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:28:37.927368705 (565h59m37.79845174s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:28:38.221000271 (565h59m38.092083316s) [upload=496233]: Done uploading segments... Dec 29 05:28:38.221019188 (565h59m38.092102223s) [upload=496233]: Upload finished. err: <nil> Duration: 6.028321922s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...X........X..X...X....X....X.........X...X...........X....] duration: 4.169877859s trace: https://jaeger.dev.storj.io/trace/28112f54d7c3878c FAIL (2025-12-29 05:33:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.169898348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..XX......X...X..XXX..X............X..X.X...XX...X.XX.......] duration: 5.689477769s trace: https://jaeger.dev.storj.io/trace/51fc923bdb532cd3 FAIL (2025-12-29 05:33:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ) context deadline exceeded ======LOGS====== c 29 05:28:34.488071392 (565h59m34.359154447s) [upload=496233]: Starting upload Dec 29 05:28:34.488166162 (565h59m34.359249217s) [upload=496233]: Done starting upload Dec 29 05:28:34.488218572 (565h59m34.359301607s) [upload=496233]: Uploading segments... Dec 29 05:28:34.488685831 (565h59m34.359768876s) [upload=496233]: Got next segment. Inline: false Dec 29 05:28:34.488733423 (565h59m34.359816458s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:28:34.488738642 (565h59m34.359821698s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:28:34.740915392 (565h59m34.611998427s) [seg_pos={0 0}, upload=496233]: Begin upload segment... Dec 29 05:28:34.740964546 (565h59m34.612047582s) [seg_pos={0 0}, upload=496233]: Done begin upload segment. Dec 29 05:28:34.740995735 (565h59m34.61207877s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741051191 (565h59m34.612134236s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741822910 (565h59m34.612905945s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742270974 (565h59m34.613354009s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742535387 (565h59m34.613618433s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742691254 (565h59m34.613774299s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743128357 (565h59m34.614211392s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743286116 (565h59m34.614369162s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743574767 (565h59m34.614657802s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743844680 (565h59m34.614927725s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743982262 (565h59m34.615065307s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744390209 (565h59m34.615473254s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744630286 (565h59m34.615713331s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744792054 (565h59m34.615875089s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745257210 (565h59m34.616340245s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745980838 (565h59m34.617063873s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:36.432947041 (565h59m36.304030076s) [upload=496233]: Next returned nil segment Dec 29 05:28:36.432985485 (565h59m36.30406854s) [upload=496233]: Waiting for error group managing segments... Dec 29 05:28:36.725517678 (565h59m36.596600713s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.728084786 (565h59m36.599167821s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.731815750 (565h59m36.602898785s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.733126446 (565h59m36.604209481s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.738813094 (565h59m36.609896129s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.744584674 (565h59m36.615667719s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.747350711 (565h59m36.618433746s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.011810150 (565h59m36.882893195s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.080270838 (565h59m36.951353883s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.087131732 (565h59m36.958214777s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.089270565 (565h59m36.9603536s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.145593362 (565h59m37.016676397s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926865657 (565h59m37.797948692s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926885936 (565h59m37.797968981s) [seg_pos={0 0}, upload=496233]: Segment reached optimal threshold of 13 pieces. Dec 29 05:28:37.927045941 (565h59m37.798128986s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection Dec 29 05:28:37.927051942 (565h59m37.798134987s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927068523 (565h59m37.798151558s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927076979 (565h59m37.798160024s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927265529 (565h59m37.798348574s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927273574 (565h59m37.798356609s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927290146 (565h59m37.798373191s) [seg_pos={0 0}, upload=496233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:28:37.927354828 (565h59m37.798437873s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:28:37.927368705 (565h59m37.79845174s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:28:38.221000271 (565h59m38.092083316s) [upload=496233]: Done uploading segments... Dec 29 05:28:38.221019188 (565h59m38.092102223s) [upload=496233]: Upload finished. err: <nil> Duration: 5.68949929s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................X.....X..X...X....................] duration: 15.435461683s trace: https://jaeger.dev.storj.io/trace/37d27399c8bd577a FAIL (2025-12-29 05:32:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 05:18:03.821720823 (565h49m3.692803878s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822185477 (565h49m3.693268522s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822572013 (565h49m3.693655058s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823071774 (565h49m3.69415482s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823476305 (565h49m3.69455936s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823922204 (565h49m3.695005249s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.824347103 (565h49m3.695430148s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.826529919 (565h49m3.697612964s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.828201432 (565h49m3.699284477s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.829987674 (565h49m3.701070719s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.831682460 (565h49m3.702765505s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.833476788 (565h49m3.704559833s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.834480228 (565h49m3.705563274s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835861419 (565h49m3.706944454s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835997778 (565h49m3.707080823s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.837791423 (565h49m3.708874468s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:05.122896476 (565h49m4.993979511s) [upload=496083]: Next returned nil segment Dec 29 05:18:05.122911676 (565h49m4.993994721s) [upload=496083]: Waiting for error group managing segments... Dec 29 05:18:05.476685321 (565h49m5.347768356s) [noise=true , address=64.49.13.26:4946 , piece_id=SLV6JONCSMPVLEW7DM7DVVOLOOSTYE2QEQWXMX2CNLA3VRNQNEZA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.686793637 (565h49m5.557876672s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.695095065 (565h49m5.56617812s) [noise=true , address=145.79.227.3:4162 , piece_id=HCXS3I6KZSTX6EAZSOBDEMVQXX7H3ML6JGNP76Z7XUOALPPCHCOQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.879055562 (565h49m5.750138627s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.952203586 (565h49m5.823286631s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.963348518 (565h49m5.834431563s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.978376062 (565h49m5.849459097s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.029930970 (565h49m5.901014005s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.064081227 (565h49m5.935164272s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.531723043 (565h49m6.402806089s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.534889613 (565h49m6.405972658s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.634538078 (565h49m6.505621123s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.635018352 (565h49m6.506101387s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664932944 (565h49m6.53601599s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664984803 (565h49m6.536067848s) [seg_pos={0 0}, upload=496083]: Segment reached optimal threshold of 14 pieces. Dec 29 05:18:06.665226433 (565h49m6.536309478s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665236311 (565h49m6.536319366s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665410774 (565h49m6.536493819s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GHRhMSQiYGrk84jej1nYkPf1A9dxxP36cPc6suGHuVALqXF4r): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665431724 (565h49m6.536514779s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665544699 (565h49m6.536627744s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection Dec 29 05:18:06.665553366 (565h49m6.536636411s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665695977 (565h49m6.536779033s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665707088 (565h49m6.536790153s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665738198 (565h49m6.536821243s) [seg_pos={0 0}, upload=496083]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:18:06.665904214 (565h49m6.536987259s) [upload=496083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:18:06.665927027 (565h49m6.537010072s) [upload=496083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:18:06.966199067 (565h49m6.837282102s) [upload=496083]: Done uploading segments... Dec 29 05:18:06.966223153 (565h49m6.837306188s) [upload=496083]: Upload finished. err: <nil> Duration: 15.43551728s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....X........X..X...X....X....X.........X...X...........X...] duration: 3.547448586s trace: https://jaeger.dev.storj.io/trace/28112f54d7c3878c FAIL (2025-12-29 05:32:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.547470028s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................X.....X..X...X...................] duration: 32.318824614s trace: https://jaeger.dev.storj.io/trace/37d27399c8bd577a FAIL (2025-12-29 05:32:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 05:18:03.821720823 (565h49m3.692803878s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822185477 (565h49m3.693268522s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822572013 (565h49m3.693655058s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823071774 (565h49m3.69415482s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823476305 (565h49m3.69455936s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823922204 (565h49m3.695005249s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.824347103 (565h49m3.695430148s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.826529919 (565h49m3.697612964s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.828201432 (565h49m3.699284477s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.829987674 (565h49m3.701070719s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.831682460 (565h49m3.702765505s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.833476788 (565h49m3.704559833s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.834480228 (565h49m3.705563274s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835861419 (565h49m3.706944454s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835997778 (565h49m3.707080823s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.837791423 (565h49m3.708874468s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:05.122896476 (565h49m4.993979511s) [upload=496083]: Next returned nil segment Dec 29 05:18:05.122911676 (565h49m4.993994721s) [upload=496083]: Waiting for error group managing segments... Dec 29 05:18:05.476685321 (565h49m5.347768356s) [noise=true , address=64.49.13.26:4946 , piece_id=SLV6JONCSMPVLEW7DM7DVVOLOOSTYE2QEQWXMX2CNLA3VRNQNEZA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.686793637 (565h49m5.557876672s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.695095065 (565h49m5.56617812s) [noise=true , address=145.79.227.3:4162 , piece_id=HCXS3I6KZSTX6EAZSOBDEMVQXX7H3ML6JGNP76Z7XUOALPPCHCOQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.879055562 (565h49m5.750138627s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.952203586 (565h49m5.823286631s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.963348518 (565h49m5.834431563s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.978376062 (565h49m5.849459097s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.029930970 (565h49m5.901014005s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.064081227 (565h49m5.935164272s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.531723043 (565h49m6.402806089s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.534889613 (565h49m6.405972658s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.634538078 (565h49m6.505621123s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.635018352 (565h49m6.506101387s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664932944 (565h49m6.53601599s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664984803 (565h49m6.536067848s) [seg_pos={0 0}, upload=496083]: Segment reached optimal threshold of 14 pieces. Dec 29 05:18:06.665226433 (565h49m6.536309478s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665236311 (565h49m6.536319366s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665410774 (565h49m6.536493819s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GHRhMSQiYGrk84jej1nYkPf1A9dxxP36cPc6suGHuVALqXF4r): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665431724 (565h49m6.536514779s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665544699 (565h49m6.536627744s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection Dec 29 05:18:06.665553366 (565h49m6.536636411s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665695977 (565h49m6.536779033s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665707088 (565h49m6.536790153s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665738198 (565h49m6.536821243s) [seg_pos={0 0}, upload=496083]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:18:06.665904214 (565h49m6.536987259s) [upload=496083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:18:06.665927027 (565h49m6.537010072s) [upload=496083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:18:06.966199067 (565h49m6.837282102s) [upload=496083]: Done uploading segments... Dec 29 05:18:06.966223153 (565h49m6.837306188s) [upload=496083]: Upload finished. err: <nil> Duration: 32.318850203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...XX......X...X..XXX..X............X..X.X...XX...X.XX......] duration: 5.861093119s trace: https://jaeger.dev.storj.io/trace/51fc923bdb532cd3 FAIL (2025-12-29 05:32:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ) context deadline exceeded ======LOGS====== c 29 05:28:34.488071392 (565h59m34.359154447s) [upload=496233]: Starting upload Dec 29 05:28:34.488166162 (565h59m34.359249217s) [upload=496233]: Done starting upload Dec 29 05:28:34.488218572 (565h59m34.359301607s) [upload=496233]: Uploading segments... Dec 29 05:28:34.488685831 (565h59m34.359768876s) [upload=496233]: Got next segment. Inline: false Dec 29 05:28:34.488733423 (565h59m34.359816458s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:28:34.488738642 (565h59m34.359821698s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:28:34.740915392 (565h59m34.611998427s) [seg_pos={0 0}, upload=496233]: Begin upload segment... Dec 29 05:28:34.740964546 (565h59m34.612047582s) [seg_pos={0 0}, upload=496233]: Done begin upload segment. Dec 29 05:28:34.740995735 (565h59m34.61207877s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741051191 (565h59m34.612134236s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741822910 (565h59m34.612905945s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742270974 (565h59m34.613354009s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742535387 (565h59m34.613618433s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742691254 (565h59m34.613774299s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743128357 (565h59m34.614211392s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743286116 (565h59m34.614369162s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743574767 (565h59m34.614657802s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743844680 (565h59m34.614927725s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743982262 (565h59m34.615065307s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744390209 (565h59m34.615473254s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744630286 (565h59m34.615713331s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744792054 (565h59m34.615875089s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745257210 (565h59m34.616340245s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745980838 (565h59m34.617063873s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:36.432947041 (565h59m36.304030076s) [upload=496233]: Next returned nil segment Dec 29 05:28:36.432985485 (565h59m36.30406854s) [upload=496233]: Waiting for error group managing segments... Dec 29 05:28:36.725517678 (565h59m36.596600713s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.728084786 (565h59m36.599167821s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.731815750 (565h59m36.602898785s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.733126446 (565h59m36.604209481s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.738813094 (565h59m36.609896129s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.744584674 (565h59m36.615667719s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.747350711 (565h59m36.618433746s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.011810150 (565h59m36.882893195s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.080270838 (565h59m36.951353883s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.087131732 (565h59m36.958214777s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.089270565 (565h59m36.9603536s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.145593362 (565h59m37.016676397s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926865657 (565h59m37.797948692s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926885936 (565h59m37.797968981s) [seg_pos={0 0}, upload=496233]: Segment reached optimal threshold of 13 pieces. Dec 29 05:28:37.927045941 (565h59m37.798128986s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection Dec 29 05:28:37.927051942 (565h59m37.798134987s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927068523 (565h59m37.798151558s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927076979 (565h59m37.798160024s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927265529 (565h59m37.798348574s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927273574 (565h59m37.798356609s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927290146 (565h59m37.798373191s) [seg_pos={0 0}, upload=496233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:28:37.927354828 (565h59m37.798437873s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:28:37.927368705 (565h59m37.79845174s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:28:38.221000271 (565h59m38.092083316s) [upload=496233]: Done uploading segments... Dec 29 05:28:38.221019188 (565h59m38.092102223s) [upload=496233]: Upload finished. err: <nil> Duration: 5.861123406s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.....X........X..X...X....X....X.........X...X...........X..] duration: 4.33592806s trace: https://jaeger.dev.storj.io/trace/28112f54d7c3878c FAIL (2025-12-29 05:32:00 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.335956713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................X.....X..X...X..................] duration: 30.673715576s trace: https://jaeger.dev.storj.io/trace/37d27399c8bd577a FAIL (2025-12-29 05:31:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 05:18:03.821720823 (565h49m3.692803878s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822185477 (565h49m3.693268522s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822572013 (565h49m3.693655058s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823071774 (565h49m3.69415482s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823476305 (565h49m3.69455936s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823922204 (565h49m3.695005249s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.824347103 (565h49m3.695430148s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.826529919 (565h49m3.697612964s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.828201432 (565h49m3.699284477s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.829987674 (565h49m3.701070719s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.831682460 (565h49m3.702765505s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.833476788 (565h49m3.704559833s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.834480228 (565h49m3.705563274s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835861419 (565h49m3.706944454s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835997778 (565h49m3.707080823s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.837791423 (565h49m3.708874468s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:05.122896476 (565h49m4.993979511s) [upload=496083]: Next returned nil segment Dec 29 05:18:05.122911676 (565h49m4.993994721s) [upload=496083]: Waiting for error group managing segments... Dec 29 05:18:05.476685321 (565h49m5.347768356s) [noise=true , address=64.49.13.26:4946 , piece_id=SLV6JONCSMPVLEW7DM7DVVOLOOSTYE2QEQWXMX2CNLA3VRNQNEZA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.686793637 (565h49m5.557876672s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.695095065 (565h49m5.56617812s) [noise=true , address=145.79.227.3:4162 , piece_id=HCXS3I6KZSTX6EAZSOBDEMVQXX7H3ML6JGNP76Z7XUOALPPCHCOQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.879055562 (565h49m5.750138627s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.952203586 (565h49m5.823286631s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.963348518 (565h49m5.834431563s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.978376062 (565h49m5.849459097s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.029930970 (565h49m5.901014005s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.064081227 (565h49m5.935164272s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.531723043 (565h49m6.402806089s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.534889613 (565h49m6.405972658s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.634538078 (565h49m6.505621123s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.635018352 (565h49m6.506101387s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664932944 (565h49m6.53601599s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664984803 (565h49m6.536067848s) [seg_pos={0 0}, upload=496083]: Segment reached optimal threshold of 14 pieces. Dec 29 05:18:06.665226433 (565h49m6.536309478s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665236311 (565h49m6.536319366s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665410774 (565h49m6.536493819s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GHRhMSQiYGrk84jej1nYkPf1A9dxxP36cPc6suGHuVALqXF4r): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665431724 (565h49m6.536514779s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665544699 (565h49m6.536627744s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection Dec 29 05:18:06.665553366 (565h49m6.536636411s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665695977 (565h49m6.536779033s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665707088 (565h49m6.536790153s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665738198 (565h49m6.536821243s) [seg_pos={0 0}, upload=496083]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:18:06.665904214 (565h49m6.536987259s) [upload=496083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:18:06.665927027 (565h49m6.537010072s) [upload=496083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:18:06.966199067 (565h49m6.837282102s) [upload=496083]: Done uploading segments... Dec 29 05:18:06.966223153 (565h49m6.837306188s) [upload=496083]: Upload finished. err: <nil> Duration: 30.67373359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....XX......X...X..XXX..X............X..X.X...XX...X.XX.....] duration: 6.02845347s trace: https://jaeger.dev.storj.io/trace/51fc923bdb532cd3 FAIL (2025-12-29 05:31:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ) context deadline exceeded ======LOGS====== c 29 05:28:34.488071392 (565h59m34.359154447s) [upload=496233]: Starting upload Dec 29 05:28:34.488166162 (565h59m34.359249217s) [upload=496233]: Done starting upload Dec 29 05:28:34.488218572 (565h59m34.359301607s) [upload=496233]: Uploading segments... Dec 29 05:28:34.488685831 (565h59m34.359768876s) [upload=496233]: Got next segment. Inline: false Dec 29 05:28:34.488733423 (565h59m34.359816458s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:28:34.488738642 (565h59m34.359821698s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:28:34.740915392 (565h59m34.611998427s) [seg_pos={0 0}, upload=496233]: Begin upload segment... Dec 29 05:28:34.740964546 (565h59m34.612047582s) [seg_pos={0 0}, upload=496233]: Done begin upload segment. Dec 29 05:28:34.740995735 (565h59m34.61207877s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741051191 (565h59m34.612134236s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741822910 (565h59m34.612905945s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742270974 (565h59m34.613354009s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742535387 (565h59m34.613618433s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742691254 (565h59m34.613774299s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743128357 (565h59m34.614211392s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743286116 (565h59m34.614369162s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743574767 (565h59m34.614657802s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743844680 (565h59m34.614927725s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743982262 (565h59m34.615065307s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744390209 (565h59m34.615473254s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744630286 (565h59m34.615713331s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744792054 (565h59m34.615875089s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745257210 (565h59m34.616340245s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745980838 (565h59m34.617063873s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:36.432947041 (565h59m36.304030076s) [upload=496233]: Next returned nil segment Dec 29 05:28:36.432985485 (565h59m36.30406854s) [upload=496233]: Waiting for error group managing segments... Dec 29 05:28:36.725517678 (565h59m36.596600713s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.728084786 (565h59m36.599167821s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.731815750 (565h59m36.602898785s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.733126446 (565h59m36.604209481s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.738813094 (565h59m36.609896129s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.744584674 (565h59m36.615667719s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.747350711 (565h59m36.618433746s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.011810150 (565h59m36.882893195s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.080270838 (565h59m36.951353883s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.087131732 (565h59m36.958214777s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.089270565 (565h59m36.9603536s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.145593362 (565h59m37.016676397s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926865657 (565h59m37.797948692s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926885936 (565h59m37.797968981s) [seg_pos={0 0}, upload=496233]: Segment reached optimal threshold of 13 pieces. Dec 29 05:28:37.927045941 (565h59m37.798128986s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection Dec 29 05:28:37.927051942 (565h59m37.798134987s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927068523 (565h59m37.798151558s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927076979 (565h59m37.798160024s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927265529 (565h59m37.798348574s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927273574 (565h59m37.798356609s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927290146 (565h59m37.798373191s) [seg_pos={0 0}, upload=496233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:28:37.927354828 (565h59m37.798437873s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:28:37.927368705 (565h59m37.79845174s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:28:38.221000271 (565h59m38.092083316s) [upload=496233]: Done uploading segments... Dec 29 05:28:38.221019188 (565h59m38.092102223s) [upload=496233]: Upload finished. err: <nil> Duration: 6.028471435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X.....X..X...X.................] duration: 49.237838819s trace: https://jaeger.dev.storj.io/trace/37d27399c8bd577a FAIL (2025-12-29 05:31:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 05:18:03.821720823 (565h49m3.692803878s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822185477 (565h49m3.693268522s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822572013 (565h49m3.693655058s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823071774 (565h49m3.69415482s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823476305 (565h49m3.69455936s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823922204 (565h49m3.695005249s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.824347103 (565h49m3.695430148s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.826529919 (565h49m3.697612964s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.828201432 (565h49m3.699284477s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.829987674 (565h49m3.701070719s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.831682460 (565h49m3.702765505s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.833476788 (565h49m3.704559833s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.834480228 (565h49m3.705563274s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835861419 (565h49m3.706944454s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835997778 (565h49m3.707080823s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.837791423 (565h49m3.708874468s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:05.122896476 (565h49m4.993979511s) [upload=496083]: Next returned nil segment Dec 29 05:18:05.122911676 (565h49m4.993994721s) [upload=496083]: Waiting for error group managing segments... Dec 29 05:18:05.476685321 (565h49m5.347768356s) [noise=true , address=64.49.13.26:4946 , piece_id=SLV6JONCSMPVLEW7DM7DVVOLOOSTYE2QEQWXMX2CNLA3VRNQNEZA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.686793637 (565h49m5.557876672s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.695095065 (565h49m5.56617812s) [noise=true , address=145.79.227.3:4162 , piece_id=HCXS3I6KZSTX6EAZSOBDEMVQXX7H3ML6JGNP76Z7XUOALPPCHCOQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.879055562 (565h49m5.750138627s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.952203586 (565h49m5.823286631s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.963348518 (565h49m5.834431563s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.978376062 (565h49m5.849459097s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.029930970 (565h49m5.901014005s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.064081227 (565h49m5.935164272s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.531723043 (565h49m6.402806089s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.534889613 (565h49m6.405972658s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.634538078 (565h49m6.505621123s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.635018352 (565h49m6.506101387s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664932944 (565h49m6.53601599s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664984803 (565h49m6.536067848s) [seg_pos={0 0}, upload=496083]: Segment reached optimal threshold of 14 pieces. Dec 29 05:18:06.665226433 (565h49m6.536309478s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665236311 (565h49m6.536319366s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665410774 (565h49m6.536493819s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GHRhMSQiYGrk84jej1nYkPf1A9dxxP36cPc6suGHuVALqXF4r): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665431724 (565h49m6.536514779s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665544699 (565h49m6.536627744s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection Dec 29 05:18:06.665553366 (565h49m6.536636411s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665695977 (565h49m6.536779033s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665707088 (565h49m6.536790153s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665738198 (565h49m6.536821243s) [seg_pos={0 0}, upload=496083]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:18:06.665904214 (565h49m6.536987259s) [upload=496083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:18:06.665927027 (565h49m6.537010072s) [upload=496083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:18:06.966199067 (565h49m6.837282102s) [upload=496083]: Done uploading segments... Dec 29 05:18:06.966223153 (565h49m6.837306188s) [upload=496083]: Upload finished. err: <nil> Duration: 49.237868507s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X.....X........X..X...X....X....X.........X...X...........X.] duration: 4.67254075s trace: https://jaeger.dev.storj.io/trace/28112f54d7c3878c FAIL (2025-12-29 05:30:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.672566259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....XX......X...X..XXX..X............X..X.X...XX...X.XX....] duration: 6.393202515s trace: https://jaeger.dev.storj.io/trace/51fc923bdb532cd3 FAIL (2025-12-29 05:30:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ) context deadline exceeded ======LOGS====== c 29 05:28:34.488071392 (565h59m34.359154447s) [upload=496233]: Starting upload Dec 29 05:28:34.488166162 (565h59m34.359249217s) [upload=496233]: Done starting upload Dec 29 05:28:34.488218572 (565h59m34.359301607s) [upload=496233]: Uploading segments... Dec 29 05:28:34.488685831 (565h59m34.359768876s) [upload=496233]: Got next segment. Inline: false Dec 29 05:28:34.488733423 (565h59m34.359816458s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:28:34.488738642 (565h59m34.359821698s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:28:34.740915392 (565h59m34.611998427s) [seg_pos={0 0}, upload=496233]: Begin upload segment... Dec 29 05:28:34.740964546 (565h59m34.612047582s) [seg_pos={0 0}, upload=496233]: Done begin upload segment. Dec 29 05:28:34.740995735 (565h59m34.61207877s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741051191 (565h59m34.612134236s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741822910 (565h59m34.612905945s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742270974 (565h59m34.613354009s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742535387 (565h59m34.613618433s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742691254 (565h59m34.613774299s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743128357 (565h59m34.614211392s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743286116 (565h59m34.614369162s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743574767 (565h59m34.614657802s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743844680 (565h59m34.614927725s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743982262 (565h59m34.615065307s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744390209 (565h59m34.615473254s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744630286 (565h59m34.615713331s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744792054 (565h59m34.615875089s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745257210 (565h59m34.616340245s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745980838 (565h59m34.617063873s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:36.432947041 (565h59m36.304030076s) [upload=496233]: Next returned nil segment Dec 29 05:28:36.432985485 (565h59m36.30406854s) [upload=496233]: Waiting for error group managing segments... Dec 29 05:28:36.725517678 (565h59m36.596600713s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.728084786 (565h59m36.599167821s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.731815750 (565h59m36.602898785s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.733126446 (565h59m36.604209481s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.738813094 (565h59m36.609896129s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.744584674 (565h59m36.615667719s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.747350711 (565h59m36.618433746s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.011810150 (565h59m36.882893195s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.080270838 (565h59m36.951353883s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.087131732 (565h59m36.958214777s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.089270565 (565h59m36.9603536s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.145593362 (565h59m37.016676397s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926865657 (565h59m37.797948692s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926885936 (565h59m37.797968981s) [seg_pos={0 0}, upload=496233]: Segment reached optimal threshold of 13 pieces. Dec 29 05:28:37.927045941 (565h59m37.798128986s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection Dec 29 05:28:37.927051942 (565h59m37.798134987s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927068523 (565h59m37.798151558s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927076979 (565h59m37.798160024s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927265529 (565h59m37.798348574s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927273574 (565h59m37.798356609s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927290146 (565h59m37.798373191s) [seg_pos={0 0}, upload=496233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:28:37.927354828 (565h59m37.798437873s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:28:37.927368705 (565h59m37.79845174s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:28:38.221000271 (565h59m38.092083316s) [upload=496233]: Done uploading segments... Dec 29 05:28:38.221019188 (565h59m38.092102223s) [upload=496233]: Upload finished. err: <nil> Duration: 6.393222323s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......XX......X...X..XXX..X............X..X.X...XX...X.XX...] duration: 5.972926095s trace: https://jaeger.dev.storj.io/trace/51fc923bdb532cd3 FAIL (2025-12-29 05:30:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ) context deadline exceeded ======LOGS====== c 29 05:28:34.488071392 (565h59m34.359154447s) [upload=496233]: Starting upload Dec 29 05:28:34.488166162 (565h59m34.359249217s) [upload=496233]: Done starting upload Dec 29 05:28:34.488218572 (565h59m34.359301607s) [upload=496233]: Uploading segments... Dec 29 05:28:34.488685831 (565h59m34.359768876s) [upload=496233]: Got next segment. Inline: false Dec 29 05:28:34.488733423 (565h59m34.359816458s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:28:34.488738642 (565h59m34.359821698s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:28:34.740915392 (565h59m34.611998427s) [seg_pos={0 0}, upload=496233]: Begin upload segment... Dec 29 05:28:34.740964546 (565h59m34.612047582s) [seg_pos={0 0}, upload=496233]: Done begin upload segment. Dec 29 05:28:34.740995735 (565h59m34.61207877s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741051191 (565h59m34.612134236s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741822910 (565h59m34.612905945s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742270974 (565h59m34.613354009s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742535387 (565h59m34.613618433s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742691254 (565h59m34.613774299s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743128357 (565h59m34.614211392s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743286116 (565h59m34.614369162s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743574767 (565h59m34.614657802s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743844680 (565h59m34.614927725s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743982262 (565h59m34.615065307s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744390209 (565h59m34.615473254s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744630286 (565h59m34.615713331s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744792054 (565h59m34.615875089s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745257210 (565h59m34.616340245s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745980838 (565h59m34.617063873s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:36.432947041 (565h59m36.304030076s) [upload=496233]: Next returned nil segment Dec 29 05:28:36.432985485 (565h59m36.30406854s) [upload=496233]: Waiting for error group managing segments... Dec 29 05:28:36.725517678 (565h59m36.596600713s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.728084786 (565h59m36.599167821s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.731815750 (565h59m36.602898785s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.733126446 (565h59m36.604209481s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.738813094 (565h59m36.609896129s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.744584674 (565h59m36.615667719s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.747350711 (565h59m36.618433746s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.011810150 (565h59m36.882893195s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.080270838 (565h59m36.951353883s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.087131732 (565h59m36.958214777s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.089270565 (565h59m36.9603536s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.145593362 (565h59m37.016676397s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926865657 (565h59m37.797948692s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926885936 (565h59m37.797968981s) [seg_pos={0 0}, upload=496233]: Segment reached optimal threshold of 13 pieces. Dec 29 05:28:37.927045941 (565h59m37.798128986s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection Dec 29 05:28:37.927051942 (565h59m37.798134987s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927068523 (565h59m37.798151558s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927076979 (565h59m37.798160024s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927265529 (565h59m37.798348574s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927273574 (565h59m37.798356609s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927290146 (565h59m37.798373191s) [seg_pos={0 0}, upload=496233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:28:37.927354828 (565h59m37.798437873s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:28:37.927368705 (565h59m37.79845174s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:28:38.221000271 (565h59m38.092083316s) [upload=496233]: Done uploading segments... Dec 29 05:28:38.221019188 (565h59m38.092102223s) [upload=496233]: Upload finished. err: <nil> Duration: 5.972953616s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................X.....X..X...X................] duration: 34.925231644s trace: https://jaeger.dev.storj.io/trace/37d27399c8bd577a FAIL (2025-12-29 05:30:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 05:18:03.821720823 (565h49m3.692803878s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822185477 (565h49m3.693268522s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822572013 (565h49m3.693655058s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823071774 (565h49m3.69415482s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823476305 (565h49m3.69455936s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823922204 (565h49m3.695005249s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.824347103 (565h49m3.695430148s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.826529919 (565h49m3.697612964s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.828201432 (565h49m3.699284477s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.829987674 (565h49m3.701070719s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.831682460 (565h49m3.702765505s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.833476788 (565h49m3.704559833s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.834480228 (565h49m3.705563274s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835861419 (565h49m3.706944454s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835997778 (565h49m3.707080823s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.837791423 (565h49m3.708874468s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:05.122896476 (565h49m4.993979511s) [upload=496083]: Next returned nil segment Dec 29 05:18:05.122911676 (565h49m4.993994721s) [upload=496083]: Waiting for error group managing segments... Dec 29 05:18:05.476685321 (565h49m5.347768356s) [noise=true , address=64.49.13.26:4946 , piece_id=SLV6JONCSMPVLEW7DM7DVVOLOOSTYE2QEQWXMX2CNLA3VRNQNEZA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.686793637 (565h49m5.557876672s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.695095065 (565h49m5.56617812s) [noise=true , address=145.79.227.3:4162 , piece_id=HCXS3I6KZSTX6EAZSOBDEMVQXX7H3ML6JGNP76Z7XUOALPPCHCOQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.879055562 (565h49m5.750138627s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.952203586 (565h49m5.823286631s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.963348518 (565h49m5.834431563s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.978376062 (565h49m5.849459097s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.029930970 (565h49m5.901014005s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.064081227 (565h49m5.935164272s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.531723043 (565h49m6.402806089s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.534889613 (565h49m6.405972658s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.634538078 (565h49m6.505621123s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.635018352 (565h49m6.506101387s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664932944 (565h49m6.53601599s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664984803 (565h49m6.536067848s) [seg_pos={0 0}, upload=496083]: Segment reached optimal threshold of 14 pieces. Dec 29 05:18:06.665226433 (565h49m6.536309478s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665236311 (565h49m6.536319366s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665410774 (565h49m6.536493819s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GHRhMSQiYGrk84jej1nYkPf1A9dxxP36cPc6suGHuVALqXF4r): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665431724 (565h49m6.536514779s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665544699 (565h49m6.536627744s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection Dec 29 05:18:06.665553366 (565h49m6.536636411s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665695977 (565h49m6.536779033s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665707088 (565h49m6.536790153s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665738198 (565h49m6.536821243s) [seg_pos={0 0}, upload=496083]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:18:06.665904214 (565h49m6.536987259s) [upload=496083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:18:06.665927027 (565h49m6.537010072s) [upload=496083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:18:06.966199067 (565h49m6.837282102s) [upload=496083]: Done uploading segments... Dec 29 05:18:06.966223153 (565h49m6.837306188s) [upload=496083]: Upload finished. err: <nil> Duration: 34.925274314s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.X.....X........X..X...X....X....X.........X...X...........X] duration: 5.001320608s trace: https://jaeger.dev.storj.io/trace/28112f54d7c3878c FAIL (2025-12-29 05:30:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001345634s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.......XX......X...X..XXX..X............X..X.X...XX...X.XX..] duration: 5.860765331s trace: https://jaeger.dev.storj.io/trace/51fc923bdb532cd3 FAIL (2025-12-29 05:29:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ) context deadline exceeded ======LOGS====== c 29 05:28:34.488071392 (565h59m34.359154447s) [upload=496233]: Starting upload Dec 29 05:28:34.488166162 (565h59m34.359249217s) [upload=496233]: Done starting upload Dec 29 05:28:34.488218572 (565h59m34.359301607s) [upload=496233]: Uploading segments... Dec 29 05:28:34.488685831 (565h59m34.359768876s) [upload=496233]: Got next segment. Inline: false Dec 29 05:28:34.488733423 (565h59m34.359816458s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:28:34.488738642 (565h59m34.359821698s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:28:34.740915392 (565h59m34.611998427s) [seg_pos={0 0}, upload=496233]: Begin upload segment... Dec 29 05:28:34.740964546 (565h59m34.612047582s) [seg_pos={0 0}, upload=496233]: Done begin upload segment. Dec 29 05:28:34.740995735 (565h59m34.61207877s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741051191 (565h59m34.612134236s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741822910 (565h59m34.612905945s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742270974 (565h59m34.613354009s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742535387 (565h59m34.613618433s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742691254 (565h59m34.613774299s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743128357 (565h59m34.614211392s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743286116 (565h59m34.614369162s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743574767 (565h59m34.614657802s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743844680 (565h59m34.614927725s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743982262 (565h59m34.615065307s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744390209 (565h59m34.615473254s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744630286 (565h59m34.615713331s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744792054 (565h59m34.615875089s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745257210 (565h59m34.616340245s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745980838 (565h59m34.617063873s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:36.432947041 (565h59m36.304030076s) [upload=496233]: Next returned nil segment Dec 29 05:28:36.432985485 (565h59m36.30406854s) [upload=496233]: Waiting for error group managing segments... Dec 29 05:28:36.725517678 (565h59m36.596600713s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.728084786 (565h59m36.599167821s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.731815750 (565h59m36.602898785s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.733126446 (565h59m36.604209481s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.738813094 (565h59m36.609896129s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.744584674 (565h59m36.615667719s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.747350711 (565h59m36.618433746s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.011810150 (565h59m36.882893195s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.080270838 (565h59m36.951353883s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.087131732 (565h59m36.958214777s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.089270565 (565h59m36.9603536s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.145593362 (565h59m37.016676397s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926865657 (565h59m37.797948692s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926885936 (565h59m37.797968981s) [seg_pos={0 0}, upload=496233]: Segment reached optimal threshold of 13 pieces. Dec 29 05:28:37.927045941 (565h59m37.798128986s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection Dec 29 05:28:37.927051942 (565h59m37.798134987s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927068523 (565h59m37.798151558s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927076979 (565h59m37.798160024s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927265529 (565h59m37.798348574s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927273574 (565h59m37.798356609s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927290146 (565h59m37.798373191s) [seg_pos={0 0}, upload=496233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:28:37.927354828 (565h59m37.798437873s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:28:37.927368705 (565h59m37.79845174s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:28:38.221000271 (565h59m38.092083316s) [upload=496233]: Done uploading segments... Dec 29 05:28:38.221019188 (565h59m38.092102223s) [upload=496233]: Upload finished. err: <nil> Duration: 5.860791862s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [........XX......X...X..XXX..X............X..X.X...XX...X.XX.] duration: 6.000577795s trace: https://jaeger.dev.storj.io/trace/51fc923bdb532cd3 FAIL (2025-12-29 05:29:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ) context deadline exceeded ======LOGS====== c 29 05:28:34.488071392 (565h59m34.359154447s) [upload=496233]: Starting upload Dec 29 05:28:34.488166162 (565h59m34.359249217s) [upload=496233]: Done starting upload Dec 29 05:28:34.488218572 (565h59m34.359301607s) [upload=496233]: Uploading segments... Dec 29 05:28:34.488685831 (565h59m34.359768876s) [upload=496233]: Got next segment. Inline: false Dec 29 05:28:34.488733423 (565h59m34.359816458s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:28:34.488738642 (565h59m34.359821698s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:28:34.740915392 (565h59m34.611998427s) [seg_pos={0 0}, upload=496233]: Begin upload segment... Dec 29 05:28:34.740964546 (565h59m34.612047582s) [seg_pos={0 0}, upload=496233]: Done begin upload segment. Dec 29 05:28:34.740995735 (565h59m34.61207877s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741051191 (565h59m34.612134236s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741822910 (565h59m34.612905945s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742270974 (565h59m34.613354009s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742535387 (565h59m34.613618433s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742691254 (565h59m34.613774299s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743128357 (565h59m34.614211392s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743286116 (565h59m34.614369162s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743574767 (565h59m34.614657802s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743844680 (565h59m34.614927725s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743982262 (565h59m34.615065307s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744390209 (565h59m34.615473254s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744630286 (565h59m34.615713331s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744792054 (565h59m34.615875089s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745257210 (565h59m34.616340245s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745980838 (565h59m34.617063873s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:36.432947041 (565h59m36.304030076s) [upload=496233]: Next returned nil segment Dec 29 05:28:36.432985485 (565h59m36.30406854s) [upload=496233]: Waiting for error group managing segments... Dec 29 05:28:36.725517678 (565h59m36.596600713s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.728084786 (565h59m36.599167821s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.731815750 (565h59m36.602898785s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.733126446 (565h59m36.604209481s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.738813094 (565h59m36.609896129s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.744584674 (565h59m36.615667719s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.747350711 (565h59m36.618433746s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.011810150 (565h59m36.882893195s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.080270838 (565h59m36.951353883s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.087131732 (565h59m36.958214777s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.089270565 (565h59m36.9603536s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.145593362 (565h59m37.016676397s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926865657 (565h59m37.797948692s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926885936 (565h59m37.797968981s) [seg_pos={0 0}, upload=496233]: Segment reached optimal threshold of 13 pieces. Dec 29 05:28:37.927045941 (565h59m37.798128986s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection Dec 29 05:28:37.927051942 (565h59m37.798134987s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927068523 (565h59m37.798151558s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927076979 (565h59m37.798160024s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927265529 (565h59m37.798348574s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927273574 (565h59m37.798356609s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927290146 (565h59m37.798373191s) [seg_pos={0 0}, upload=496233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:28:37.927354828 (565h59m37.798437873s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:28:37.927368705 (565h59m37.79845174s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:28:38.221000271 (565h59m38.092083316s) [upload=496233]: Done uploading segments... Dec 29 05:28:38.221019188 (565h59m38.092102223s) [upload=496233]: Upload finished. err: <nil> Duration: 6.0006066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................X.....X..X...X...............] duration: 42.448518872s trace: https://jaeger.dev.storj.io/trace/37d27399c8bd577a FAIL (2025-12-29 05:29:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 05:18:03.821720823 (565h49m3.692803878s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822185477 (565h49m3.693268522s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822572013 (565h49m3.693655058s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823071774 (565h49m3.69415482s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823476305 (565h49m3.69455936s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823922204 (565h49m3.695005249s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.824347103 (565h49m3.695430148s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.826529919 (565h49m3.697612964s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.828201432 (565h49m3.699284477s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.829987674 (565h49m3.701070719s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.831682460 (565h49m3.702765505s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.833476788 (565h49m3.704559833s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.834480228 (565h49m3.705563274s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835861419 (565h49m3.706944454s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835997778 (565h49m3.707080823s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.837791423 (565h49m3.708874468s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:05.122896476 (565h49m4.993979511s) [upload=496083]: Next returned nil segment Dec 29 05:18:05.122911676 (565h49m4.993994721s) [upload=496083]: Waiting for error group managing segments... Dec 29 05:18:05.476685321 (565h49m5.347768356s) [noise=true , address=64.49.13.26:4946 , piece_id=SLV6JONCSMPVLEW7DM7DVVOLOOSTYE2QEQWXMX2CNLA3VRNQNEZA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.686793637 (565h49m5.557876672s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.695095065 (565h49m5.56617812s) [noise=true , address=145.79.227.3:4162 , piece_id=HCXS3I6KZSTX6EAZSOBDEMVQXX7H3ML6JGNP76Z7XUOALPPCHCOQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.879055562 (565h49m5.750138627s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.952203586 (565h49m5.823286631s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.963348518 (565h49m5.834431563s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.978376062 (565h49m5.849459097s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.029930970 (565h49m5.901014005s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.064081227 (565h49m5.935164272s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.531723043 (565h49m6.402806089s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.534889613 (565h49m6.405972658s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.634538078 (565h49m6.505621123s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.635018352 (565h49m6.506101387s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664932944 (565h49m6.53601599s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664984803 (565h49m6.536067848s) [seg_pos={0 0}, upload=496083]: Segment reached optimal threshold of 14 pieces. Dec 29 05:18:06.665226433 (565h49m6.536309478s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665236311 (565h49m6.536319366s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665410774 (565h49m6.536493819s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GHRhMSQiYGrk84jej1nYkPf1A9dxxP36cPc6suGHuVALqXF4r): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665431724 (565h49m6.536514779s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665544699 (565h49m6.536627744s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection Dec 29 05:18:06.665553366 (565h49m6.536636411s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665695977 (565h49m6.536779033s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665707088 (565h49m6.536790153s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665738198 (565h49m6.536821243s) [seg_pos={0 0}, upload=496083]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:18:06.665904214 (565h49m6.536987259s) [upload=496083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:18:06.665927027 (565h49m6.537010072s) [upload=496083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:18:06.966199067 (565h49m6.837282102s) [upload=496083]: Done uploading segments... Dec 29 05:18:06.966223153 (565h49m6.837306188s) [upload=496083]: Upload finished. err: <nil> Duration: 42.448537468s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..X.....X........X..X...X....X....X.........X...X...........] duration: 3.379698944s trace: https://jaeger.dev.storj.io/trace/159c3b7a9a64e4e4 FAIL (2025-12-29 05:29:25 UTC) orbiter warning: context deadline exceeded Duration: 3.379758909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X........XX......X...X..XXX..X............X..X.X...XX...X.XX] duration: 30.002532051s trace: https://jaeger.dev.storj.io/trace/51fc923bdb532cd3 FAIL (2025-12-29 05:29:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ) context deadline exceeded ======LOGS====== c 29 05:28:34.488071392 (565h59m34.359154447s) [upload=496233]: Starting upload Dec 29 05:28:34.488166162 (565h59m34.359249217s) [upload=496233]: Done starting upload Dec 29 05:28:34.488218572 (565h59m34.359301607s) [upload=496233]: Uploading segments... Dec 29 05:28:34.488685831 (565h59m34.359768876s) [upload=496233]: Got next segment. Inline: false Dec 29 05:28:34.488733423 (565h59m34.359816458s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:28:34.488738642 (565h59m34.359821698s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:28:34.740915392 (565h59m34.611998427s) [seg_pos={0 0}, upload=496233]: Begin upload segment... Dec 29 05:28:34.740964546 (565h59m34.612047582s) [seg_pos={0 0}, upload=496233]: Done begin upload segment. Dec 29 05:28:34.740995735 (565h59m34.61207877s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741051191 (565h59m34.612134236s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.741822910 (565h59m34.612905945s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742270974 (565h59m34.613354009s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742535387 (565h59m34.613618433s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.742691254 (565h59m34.613774299s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743128357 (565h59m34.614211392s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743286116 (565h59m34.614369162s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743574767 (565h59m34.614657802s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743844680 (565h59m34.614927725s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.743982262 (565h59m34.615065307s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744390209 (565h59m34.615473254s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744630286 (565h59m34.615713331s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.744792054 (565h59m34.615875089s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745257210 (565h59m34.616340245s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:34.745980838 (565h59m34.617063873s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Uploading piece... Dec 29 05:28:36.432947041 (565h59m36.304030076s) [upload=496233]: Next returned nil segment Dec 29 05:28:36.432985485 (565h59m36.30406854s) [upload=496233]: Waiting for error group managing segments... Dec 29 05:28:36.725517678 (565h59m36.596600713s) [noise=true , address=145.79.227.15:4875 , piece_id=2PEIA62YP5S5XNRFRZ7AM4TJBLZ75UY2Y53GLKFSXFYXTZ4PUCUA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.728084786 (565h59m36.599167821s) [noise=true , address=145.79.227.16:4912 , piece_id=UZNU5PXKSVTVSOKL67JPTQZXWM6264MDA2HHDOBH3AZ2DIPXNFGA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.731815750 (565h59m36.602898785s) [noise=true , address=145.79.227.11:4730 , piece_id=DYJDQ5IASH6ODFS7HSGLUNK7YXC45J2SJKPKIRHPS4RYJYF2KIEQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.733126446 (565h59m36.604209481s) [noise=true , address=145.79.227.9:4367 , piece_id=ZTTNALWJUCO4EXXTA3A5JEHIZ6Q5I5NVIOT4HEPXQ5C3GZKZKY2Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.738813094 (565h59m36.609896129s) [noise=true , address=145.79.227.42:4123 , piece_id=BETADL5ZDLWNKROPXWQZSYQ6Z2PQFBVRKWTZHE6RPVZ2S5HG3PFA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.744584674 (565h59m36.615667719s) [noise=true , address=145.79.227.6:4259 , piece_id=HBX626QUOO7I7EZ7PTFJ2AHE4QBACHO7HJH5AJPE2AL7ELDRHACA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:36.747350711 (565h59m36.618433746s) [noise=true , address=145.79.227.19:4077 , piece_id=E2RXYQKLQTGHGDTZGTP6NU55IKCNIZYGAH7FM5PTLON3FVQHV76Q, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.011810150 (565h59m36.882893195s) [noise=true , address=77.90.31.14:4664 , piece_id=6Y6GNPJEGKZJYBZ72SGPA54VSGEZJ4BOYOF3WNNYF7VYUKIKYKJA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.080270838 (565h59m36.951353883s) [noise=true , address=45.91.116.10:4082 , piece_id=K5GDJ524WY4EDNFPUZTZEMZMDY7QT32SN5S6U24TQTNKNDCJJZHQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.087131732 (565h59m36.958214777s) [noise=true , address=45.91.116.14:4189 , piece_id=BPIKYNSZEU7GIEX76UYFY2PAOH5P7Z2P52A534VYRYDL3Z3VBZQQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.089270565 (565h59m36.9603536s) [noise=true , address=45.91.116.9:4262 , piece_id=6YKFZCUMESHX3NKAWIUPQTXY4T4D4M5PIZ64YWRX4ZQM6DCDPQDQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.145593362 (565h59m37.016676397s) [noise=true , address=45.91.116.5:4300 , piece_id=GB32S7UNZNXYQA436L6ELB6ABJRQZ3OPXTCT3HFPMMRKLJ7ZJPLQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926865657 (565h59m37.797948692s) [noise=true , address=145.79.227.43:4157 , piece_id=P3S4JU26GHBOYSEACAO3ECGEAMCJZQ6NSCV5SG3OV4MXSTUOTSUQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: <nil> Dec 29 05:28:37.926885936 (565h59m37.797968981s) [seg_pos={0 0}, upload=496233]: Segment reached optimal threshold of 13 pieces. Dec 29 05:28:37.927045941 (565h59m37.798128986s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57082->77.90.31.11:4554: use of closed network connection Dec 29 05:28:37.927051942 (565h59m37.798134987s) [noise=true , address=77.90.31.11:4554 , piece_id=NEUJI5PW2DDKWDNLHW7TT74MXDYMFXKV2ORLCAPOTGSZCOYTSQ3A, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927068523 (565h59m37.798151558s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927076979 (565h59m37.798160024s) [noise=true , address=145.79.227.40:4051 , piece_id=U2FQZEAZ2H4LB3MRHCCV3EQVMICWJ67G5ANCAMH6ZSOG3U2FM2SA, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927265529 (565h59m37.798348574s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 29 05:28:37.927273574 (565h59m37.798356609s) [noise=true , address=77.90.31.19:4844 , piece_id=7E4V6ZN45DR4HZZF4K5VAPIIWN4UZQBGCKKTIKRRLHNCEDMYFLAQ, seg_pos={0 0}, upload=496233]: Long tail cancellation due to optimal threshold Dec 29 05:28:37.927290146 (565h59m37.798373191s) [seg_pos={0 0}, upload=496233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:28:37.927354828 (565h59m37.798437873s) [upload=496233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:28:37.927368705 (565h59m37.79845174s) [upload=496233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:28:38.221000271 (565h59m38.092083316s) [upload=496233]: Done uploading segments... Dec 29 05:28:38.221019188 (565h59m38.092102223s) [upload=496233]: Upload finished. err: <nil> Duration: 30.002580603s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................X.....X..X...X..............] duration: 33.836414401s trace: https://jaeger.dev.storj.io/trace/37d27399c8bd577a FAIL (2025-12-29 05:28:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 05:18:03.821720823 (565h49m3.692803878s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822185477 (565h49m3.693268522s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822572013 (565h49m3.693655058s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823071774 (565h49m3.69415482s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823476305 (565h49m3.69455936s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823922204 (565h49m3.695005249s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.824347103 (565h49m3.695430148s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.826529919 (565h49m3.697612964s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.828201432 (565h49m3.699284477s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.829987674 (565h49m3.701070719s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.831682460 (565h49m3.702765505s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.833476788 (565h49m3.704559833s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.834480228 (565h49m3.705563274s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835861419 (565h49m3.706944454s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835997778 (565h49m3.707080823s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.837791423 (565h49m3.708874468s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:05.122896476 (565h49m4.993979511s) [upload=496083]: Next returned nil segment Dec 29 05:18:05.122911676 (565h49m4.993994721s) [upload=496083]: Waiting for error group managing segments... Dec 29 05:18:05.476685321 (565h49m5.347768356s) [noise=true , address=64.49.13.26:4946 , piece_id=SLV6JONCSMPVLEW7DM7DVVOLOOSTYE2QEQWXMX2CNLA3VRNQNEZA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.686793637 (565h49m5.557876672s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.695095065 (565h49m5.56617812s) [noise=true , address=145.79.227.3:4162 , piece_id=HCXS3I6KZSTX6EAZSOBDEMVQXX7H3ML6JGNP76Z7XUOALPPCHCOQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.879055562 (565h49m5.750138627s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.952203586 (565h49m5.823286631s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.963348518 (565h49m5.834431563s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.978376062 (565h49m5.849459097s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.029930970 (565h49m5.901014005s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.064081227 (565h49m5.935164272s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.531723043 (565h49m6.402806089s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.534889613 (565h49m6.405972658s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.634538078 (565h49m6.505621123s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.635018352 (565h49m6.506101387s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664932944 (565h49m6.53601599s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664984803 (565h49m6.536067848s) [seg_pos={0 0}, upload=496083]: Segment reached optimal threshold of 14 pieces. Dec 29 05:18:06.665226433 (565h49m6.536309478s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665236311 (565h49m6.536319366s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665410774 (565h49m6.536493819s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GHRhMSQiYGrk84jej1nYkPf1A9dxxP36cPc6suGHuVALqXF4r): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665431724 (565h49m6.536514779s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665544699 (565h49m6.536627744s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection Dec 29 05:18:06.665553366 (565h49m6.536636411s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665695977 (565h49m6.536779033s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665707088 (565h49m6.536790153s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665738198 (565h49m6.536821243s) [seg_pos={0 0}, upload=496083]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:18:06.665904214 (565h49m6.536987259s) [upload=496083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:18:06.665927027 (565h49m6.537010072s) [upload=496083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:18:06.966199067 (565h49m6.837282102s) [upload=496083]: Done uploading segments... Dec 29 05:18:06.966223153 (565h49m6.837306188s) [upload=496083]: Upload finished. err: <nil> Duration: 33.836442995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX........XX......X...X..XXX..X............X..X.X...XX...X.X] duration: 30.002768393s trace: https://jaeger.dev.storj.io/trace/10faa1ac0e990533 FAIL (2025-12-29 05:28:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: BWV7LYX4AF2J4ISFIHEQWDZVTBLS7ULDMXHKD25X3ECHUBJMBRAA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: XOVILHDQZ3C7CQNDC6CEYRIN64HFVAHDGI37X3X3LQBNUFEPVZ4Q) context deadline exceeded ======LOGS====== 29 05:27:52.827020934 (565h58m52.698103979s) [upload=496226]: Starting upload Dec 29 05:27:52.827080947 (565h58m52.698163982s) [upload=496226]: Done starting upload Dec 29 05:27:52.827215053 (565h58m52.698298098s) [upload=496226]: Uploading segments... Dec 29 05:27:52.827264618 (565h58m52.698347663s) [upload=496226]: Got next segment. Inline: false Dec 29 05:27:52.827310264 (565h58m52.698393309s) [upload=496226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:27:52.827315254 (565h58m52.698398289s) [upload=496226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:27:53.073944190 (565h58m52.945027235s) [seg_pos={0 0}, upload=496226]: Begin upload segment... Dec 29 05:27:53.073984216 (565h58m52.945067261s) [seg_pos={0 0}, upload=496226]: Done begin upload segment. Dec 29 05:27:53.074008272 (565h58m52.945091307s) [noise=true , address=45.91.116.5:4297 , piece_id=XOVILHDQZ3C7CQNDC6CEYRIN64HFVAHDGI37X3X3LQBNUFEPVZ4Q, seg_pos={0 0}, upload=496226]: Uploading piece... Dec 29 05:27:53.074016598 (565h58m52.945099633s) [noise=true , address=45.91.116.7:4010 , piece_id=CZPYCRHESSCM7OWMJXVRBUGIJP4DM7THWPBKQ3R4DXGWYSI6RWIA, seg_pos={0 0}, upload=496226]: Uploading piece... Dec 29 05:27:53.074591303 (565h58m52.945674348s) [noise=true , address=45.91.116.13:4156 , piece_id=H65YGG5J5GEC3TJ3JM7MTEK4QAOEOPXXKZANJ6ZDXRMJNHIVYGUQ, seg_pos={0 0}, upload=496226]: Uploading piece... Dec 29 05:27:53.075082398 (565h58m52.946165433s) [noise=true , address=145.79.227.26:4655 , piece_id=JKBYMYJ6NGYXLOET76LS6DRFZOQOIH3RP67BTQGKKX5GKDBSWBPA, seg_pos={0 0}, upload=496226]: Uploading piece... Dec 29 05:27:53.075524018 (565h58m52.946607053s) [noise=true , address=145.79.227.50:4374 , piece_id=YWBGMRH4IKIB5ZMATX6W5LNH44EXFU77E3BI6NKODKDNXL5QDJZQ, seg_pos={0 0}, upload=496226]: Uploading piece... Dec 29 05:27:53.075961882 (565h58m52.947044917s) [noise=true , address=77.90.31.10:4517 , piece_id=BWV7LYX4AF2J4ISFIHEQWDZVTBLS7ULDMXHKD25X3ECHUBJMBRAA, seg_pos={0 0}, upload=496226]: Uploading piece... Dec 29 05:27:53.076298634 (565h58m52.947381669s) [noise=true , address=145.79.227.15:4876 , piece_id=UIRLY7LWVCR6UINVLIW6MXU7UDCNUUZ27RP2J2IL6K2UV2ABO2PQ, seg_pos={0 0}, upload=496226]: Uploading piece... Dec 29 05:27:53.076603734 (565h58m52.947686779s) [noise=true , address=145.79.227.6:4259 , piece_id=CKTUPFMJ6ZZ4MCO7NX6MF2K6IMAOYQTVC7DRDNRCK5LN4KZIZUVA, seg_pos={0 0}, upload=496226]: Uploading piece... Dec 29 05:27:53.076952388 (565h58m52.948035433s) [noise=true , address=145.79.227.17:4972 , piece_id=F2A2INC3PURSOW7IOR3RSBM3DKPHGTB62TNIDX4PZSYKPN4DSQRA, seg_pos={0 0}, upload=496226]: Uploading piece... Dec 29 05:27:53.077335026 (565h58m52.948418061s) [noise=true , address=145.79.227.32:4835 , piece_id=WF7DO7RAVTZVT576CWJAVBUTQTBJGI5B77G7AACRW4SGRAS3BMDA, seg_pos={0 0}, upload=496226]: Uploading piece... Dec 29 05:27:53.078099963 (565h58m52.949182998s) [noise=true , address=77.90.31.21:4878 , piece_id=YPLVRCWJ4YB7QTBWMGBY73DJFC6LZNYONPNXMUUXXAGFKYUO4FGQ, seg_pos={0 0}, upload=496226]: Uploading piece... Dec 29 05:27:53.078532778 (565h58m52.949615813s) [noise=true , address=145.79.227.47:4265 , piece_id=EG3BLKVPIEJWDIFMOTH7KIHU4PZUDAMR27NYPWUFP4624X3PBEFQ, seg_pos={0 0}, upload=496226]: Uploading piece... Dec 29 05:27:53.079118342 (565h58m52.950201377s) [noise=true , address=145.79.227.51:4409 , piece_id=RJGISXM5H6O254WTERMZUWVW7FPCLCZ2FJ37LPPEEL4BLVVYVJSQ, seg_pos={0 0}, upload=496226]: Uploading piece... Dec 29 05:27:53.079627051 (565h58m52.950710086s) [noise=true , address=145.79.227.52:4446 , piece_id=B3BJCELAGXA4E7WVJPYTHDWADQ5XFEWF7542MJDXUHGQG4MCDJ5A, seg_pos={0 0}, upload=496226]: Uploading piece... Dec 29 05:27:53.080039176 (565h58m52.951122211s) [noise=true , address=145.79.227.45:4231 , piece_id=QVTU3I5UISH22IJ5E4PCWH26C5VKIKPR7HSG5TWI2OMM32TCMENA, seg_pos={0 0}, upload=496226]: Uploading piece... Dec 29 05:27:53.081581112 (565h58m52.952664147s) [noise=true , address=45.91.116.8:4045 , piece_id=YEZATJBICFVTH7AZIPWPCTRJ6F2ZFGOSHU7HKQSWXGYCVEJCO4WA, seg_pos={0 0}, upload=496226]: Uploading piece... Dec 29 05:27:54.784953544 (565h58m54.656593881s) [upload=496226]: Next returned nil segment Dec 29 05:27:54.785697201 (565h58m54.656780246s) [upload=496226]: Waiting for error group managing segments... Dec 29 05:27:55.065015053 (565h58m54.936098088s) [noise=true , address=145.79.227.6:4259 , piece_id=CKTUPFMJ6ZZ4MCO7NX6MF2K6IMAOYQTVC7DRDNRCK5LN4KZIZUVA, seg_pos={0 0}, upload=496226]: Done uploading piece. err: <nil> Dec 29 05:27:55.068116327 (565h58m54.939199362s) [noise=true , address=145.79.227.32:4835 , piece_id=WF7DO7RAVTZVT576CWJAVBUTQTBJGI5B77G7AACRW4SGRAS3BMDA, seg_pos={0 0}, upload=496226]: Done uploading piece. err: <nil> Dec 29 05:27:55.068883858 (565h58m54.939966893s) [noise=true , address=145.79.227.51:4409 , piece_id=RJGISXM5H6O254WTERMZUWVW7FPCLCZ2FJ37LPPEEL4BLVVYVJSQ, seg_pos={0 0}, upload=496226]: Done uploading piece. err: <nil> Dec 29 05:27:55.070565621 (565h58m54.941648677s) [noise=true , address=145.79.227.17:4972 , piece_id=F2A2INC3PURSOW7IOR3RSBM3DKPHGTB62TNIDX4PZSYKPN4DSQRA, seg_pos={0 0}, upload=496226]: Done uploading piece. err: <nil> Dec 29 05:27:55.074039405 (565h58m54.94512245s) [noise=true , address=145.79.227.45:4231 , piece_id=QVTU3I5UISH22IJ5E4PCWH26C5VKIKPR7HSG5TWI2OMM32TCMENA, seg_pos={0 0}, upload=496226]: Done uploading piece. err: <nil> Dec 29 05:27:55.083222261 (565h58m54.954305306s) [noise=true , address=145.79.227.26:4655 , piece_id=JKBYMYJ6NGYXLOET76LS6DRFZOQOIH3RP67BTQGKKX5GKDBSWBPA, seg_pos={0 0}, upload=496226]: Done uploading piece. err: <nil> Dec 29 05:27:55.090023993 (565h58m54.961107048s) [noise=true , address=145.79.227.50:4374 , piece_id=YWBGMRH4IKIB5ZMATX6W5LNH44EXFU77E3BI6NKODKDNXL5QDJZQ, seg_pos={0 0}, upload=496226]: Done uploading piece. err: <nil> Dec 29 05:27:55.096320332 (565h58m54.967403377s) [noise=true , address=145.79.227.47:4265 , piece_id=EG3BLKVPIEJWDIFMOTH7KIHU4PZUDAMR27NYPWUFP4624X3PBEFQ, seg_pos={0 0}, upload=496226]: Done uploading piece. err: <nil> Dec 29 05:27:55.103073702 (565h58m54.974156737s) [noise=true , address=145.79.227.15:4876 , piece_id=UIRLY7LWVCR6UINVLIW6MXU7UDCNUUZ27RP2J2IL6K2UV2ABO2PQ, seg_pos={0 0}, upload=496226]: Done uploading piece. err: <nil> Dec 29 05:27:55.130431575 (565h58m55.00151462s) [noise=true , address=145.79.227.52:4446 , piece_id=B3BJCELAGXA4E7WVJPYTHDWADQ5XFEWF7542MJDXUHGQG4MCDJ5A, seg_pos={0 0}, upload=496226]: Done uploading piece. err: <nil> Dec 29 05:27:55.265046841 (565h58m55.136129877s) [noise=true , address=77.90.31.10:4517 , piece_id=BWV7LYX4AF2J4ISFIHEQWDZVTBLS7ULDMXHKD25X3ECHUBJMBRAA, seg_pos={0 0}, upload=496226]: Done uploading piece. err: <nil> Dec 29 05:27:55.410504574 (565h58m55.281587609s) [noise=true , address=45.91.116.8:4045 , piece_id=YEZATJBICFVTH7AZIPWPCTRJ6F2ZFGOSHU7HKQSWXGYCVEJCO4WA, seg_pos={0 0}, upload=496226]: Done uploading piece. err: <nil> Dec 29 05:27:55.448600003 (565h58m55.319683048s) [noise=true , address=45.91.116.5:4297 , piece_id=XOVILHDQZ3C7CQNDC6CEYRIN64HFVAHDGI37X3X3LQBNUFEPVZ4Q, seg_pos={0 0}, upload=496226]: Done uploading piece. err: <nil> Dec 29 05:27:55.448624640 (565h58m55.319707685s) [seg_pos={0 0}, upload=496226]: Segment reached optimal threshold of 13 pieces. Dec 29 05:27:55.448840531 (565h58m55.319923576s) [noise=true , address=77.90.31.21:4878 , piece_id=YPLVRCWJ4YB7QTBWMGBY73DJFC6LZNYONPNXMUUXXAGFKYUO4FGQ, seg_pos={0 0}, upload=496226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:55224->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:55224->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55224->77.90.31.21:4878: use of closed network connection Dec 29 05:27:55.448849087 (565h58m55.319932122s) [noise=true , address=77.90.31.21:4878 , piece_id=YPLVRCWJ4YB7QTBWMGBY73DJFC6LZNYONPNXMUUXXAGFKYUO4FGQ, seg_pos={0 0}, upload=496226]: Long tail cancellation due to optimal threshold Dec 29 05:27:55.448986288 (565h58m55.320069323s) [noise=true , address=45.91.116.13:4156 , piece_id=H65YGG5J5GEC3TJ3JM7MTEK4QAOEOPXXKZANJ6ZDXRMJNHIVYGUQ, seg_pos={0 0}, upload=496226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 05:27:55.449003111 (565h58m55.320086146s) [noise=true , address=45.91.116.13:4156 , piece_id=H65YGG5J5GEC3TJ3JM7MTEK4QAOEOPXXKZANJ6ZDXRMJNHIVYGUQ, seg_pos={0 0}, upload=496226]: Long tail cancellation due to optimal threshold Dec 29 05:27:55.449164106 (565h58m55.320247152s) [noise=true , address=45.91.116.7:4010 , piece_id=CZPYCRHESSCM7OWMJXVRBUGIJP4DM7THWPBKQ3R4DXGWYSI6RWIA, seg_pos={0 0}, upload=496226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 05:27:55.449173916 (565h58m55.320256961s) [noise=true , address=45.91.116.7:4010 , piece_id=CZPYCRHESSCM7OWMJXVRBUGIJP4DM7THWPBKQ3R4DXGWYSI6RWIA, seg_pos={0 0}, upload=496226]: Long tail cancellation due to optimal threshold Dec 29 05:27:55.449194905 (565h58m55.32027795s) [seg_pos={0 0}, upload=496226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:27:55.449281150 (565h58m55.320364185s) [upload=496226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:27:55.449287652 (565h58m55.320370687s) [upload=496226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:27:55.747960348 (565h58m55.619043383s) [upload=496226]: Done uploading segments... Dec 29 05:27:55.747984162 (565h58m55.619067197s) [upload=496226]: Upload finished. err: <nil> Duration: 30.002789854s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...X.....X........X..X...X....X....X.........X...X..........] duration: 3.65356708s trace: https://jaeger.dev.storj.io/trace/159c3b7a9a64e4e4 FAIL (2025-12-29 05:28:19 UTC) orbiter warning: context deadline exceeded Duration: 3.653594041s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................X.....X..X...X.............] duration: 15.821569283s trace: https://jaeger.dev.storj.io/trace/37d27399c8bd577a FAIL (2025-12-29 05:27:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 05:18:03.821720823 (565h49m3.692803878s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822185477 (565h49m3.693268522s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822572013 (565h49m3.693655058s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823071774 (565h49m3.69415482s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823476305 (565h49m3.69455936s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823922204 (565h49m3.695005249s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.824347103 (565h49m3.695430148s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.826529919 (565h49m3.697612964s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.828201432 (565h49m3.699284477s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.829987674 (565h49m3.701070719s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.831682460 (565h49m3.702765505s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.833476788 (565h49m3.704559833s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.834480228 (565h49m3.705563274s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835861419 (565h49m3.706944454s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835997778 (565h49m3.707080823s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.837791423 (565h49m3.708874468s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:05.122896476 (565h49m4.993979511s) [upload=496083]: Next returned nil segment Dec 29 05:18:05.122911676 (565h49m4.993994721s) [upload=496083]: Waiting for error group managing segments... Dec 29 05:18:05.476685321 (565h49m5.347768356s) [noise=true , address=64.49.13.26:4946 , piece_id=SLV6JONCSMPVLEW7DM7DVVOLOOSTYE2QEQWXMX2CNLA3VRNQNEZA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.686793637 (565h49m5.557876672s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.695095065 (565h49m5.56617812s) [noise=true , address=145.79.227.3:4162 , piece_id=HCXS3I6KZSTX6EAZSOBDEMVQXX7H3ML6JGNP76Z7XUOALPPCHCOQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.879055562 (565h49m5.750138627s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.952203586 (565h49m5.823286631s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.963348518 (565h49m5.834431563s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.978376062 (565h49m5.849459097s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.029930970 (565h49m5.901014005s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.064081227 (565h49m5.935164272s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.531723043 (565h49m6.402806089s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.534889613 (565h49m6.405972658s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.634538078 (565h49m6.505621123s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.635018352 (565h49m6.506101387s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664932944 (565h49m6.53601599s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664984803 (565h49m6.536067848s) [seg_pos={0 0}, upload=496083]: Segment reached optimal threshold of 14 pieces. Dec 29 05:18:06.665226433 (565h49m6.536309478s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665236311 (565h49m6.536319366s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665410774 (565h49m6.536493819s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GHRhMSQiYGrk84jej1nYkPf1A9dxxP36cPc6suGHuVALqXF4r): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665431724 (565h49m6.536514779s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665544699 (565h49m6.536627744s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection Dec 29 05:18:06.665553366 (565h49m6.536636411s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665695977 (565h49m6.536779033s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665707088 (565h49m6.536790153s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665738198 (565h49m6.536821243s) [seg_pos={0 0}, upload=496083]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:18:06.665904214 (565h49m6.536987259s) [upload=496083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:18:06.665927027 (565h49m6.537010072s) [upload=496083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:18:06.966199067 (565h49m6.837282102s) [upload=496083]: Done uploading segments... Dec 29 05:18:06.966223153 (565h49m6.837306188s) [upload=496083]: Upload finished. err: <nil> Duration: 15.821599541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXX........XX......X...X..XXX..X............X..X.X...XX...X.] duration: 5.697269803s trace: https://jaeger.dev.storj.io/trace/741e8c2408b5518b FAIL (2025-12-29 05:27:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: DJBZUKDHWYTI7GNW6LZRKOCLSVT6RN6DV57EQJBCIWLGGA6RDS4A) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: EQONRO6RLYNKQFEEFIIMOCMGEVW34KORU5FW2PJ2JN44C3DRLE5A) context deadline exceeded ======LOGS====== Dec 29 05:26:37.406535872 (565h57m37.277618907s) [upload=496206]: Starting upload Dec 29 05:26:37.406694414 (565h57m37.277777459s) [upload=496206]: Done starting upload Dec 29 05:26:37.406964789 (565h57m37.278047824s) [upload=496206]: Uploading segments... Dec 29 05:26:37.407065891 (565h57m37.278148936s) [upload=496206]: Got next segment. Inline: false Dec 29 05:26:37.407106669 (565h57m37.278189704s) [upload=496206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:26:37.407113131 (565h57m37.278196176s) [upload=496206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:26:37.652478764 (565h57m37.523561829s) [seg_pos={0 0}, upload=496206]: Begin upload segment... Dec 29 05:26:37.652535191 (565h57m37.523618236s) [seg_pos={0 0}, upload=496206]: Done begin upload segment. Dec 29 05:26:37.652566821 (565h57m37.523649866s) [noise=true , address=77.90.31.21:4877 , piece_id=KRXZW7YVA4HX7LDJQA3YSMU6SX5H2XKKYJDI3JIZYNL5BMN367SQ, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.652597229 (565h57m37.523680264s) [noise=true , address=145.79.227.15:4874 , piece_id=V2ABJ6CM3ISMEVMENDXZAY7NIEVBZFWVKN4FSNHHXY36GJEARXHA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.653074829 (565h57m37.524157874s) [noise=true , address=77.90.31.11:4554 , piece_id=ML7UIMM4HDYWUWUW7KNNAADGLWMVBWOHO32IONXOVJBJJL55ZUEA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.653468719 (565h57m37.524551764s) [noise=true , address=77.90.31.19:4842 , piece_id=EQONRO6RLYNKQFEEFIIMOCMGEVW34KORU5FW2PJ2JN44C3DRLE5A, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.653909428 (565h57m37.524992473s) [noise=true , address=45.91.116.14:4190 , piece_id=GN26T6PWDQFMD7E5SGQO4RP7IJYBJ5CPHWJZUDAHPHAOIBJYMN2A, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.654336371 (565h57m37.525419406s) [noise=true , address=145.79.227.41:4087 , piece_id=UOUP4RQNZJ2SR2SZX2LKBAHZIS3PAM2DYDZ6JESKNUMBNRB3TEDA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.654745650 (565h57m37.525828685s) [noise=true , address=145.79.227.36:4518 , piece_id=JQCTLGBLLQ5E7LVRYYUF2DCSPT52CTR6IM5NGNKSAZU2NGD4N7JA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.655196860 (565h57m37.526279905s) [noise=true , address=145.79.227.45:4231 , piece_id=7Q6ZPQP3AE35XEZS4WLDCN44FDWZ4TXQDDQ432H64VKEP4QVHETA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.655732279 (565h57m37.526815324s) [noise=true , address=45.91.116.10:4083 , piece_id=LROVVOUI4LK63HJ7UKVKUNSQV6CISQKZA4DWCTHDLFCFUH2GGGHA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.656260625 (565h57m37.52734367s) [noise=true , address=45.91.116.6:4973 , piece_id=GMT2WKZNH6YZSG5VJQO4CEQUYQO4QEXLPDYPFFLJDE4HCXY37KZA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.657867896 (565h57m37.528950941s) [noise=true , address=145.79.227.18:4007 , piece_id=SGEFJP7W6JGRKHOPS5VRKICY2EWN7E45LGG6NEYWBICF7FW37KBA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.659488141 (565h57m37.530571186s) [noise=true , address=45.91.116.9:4261 , piece_id=HU4R532QDQTYGUVKGXV5RERNQ2Y565J7IZBXNPT4O3QLK7GDQVVA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.661071716 (565h57m37.532154761s) [noise=true , address=45.91.116.7:4010 , piece_id=4DIMMGXSMZNYLKV7TCBD34YSPWIWE6EMMJTZIKWALDJH5VS4PUWA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.662928932 (565h57m37.534011977s) [noise=true , address=145.79.227.11:4729 , piece_id=KNS5P5K2TSM6LH6A574VLC74EMTJLVCITRRINSV5JSHBMVMHGL5Q, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.664830573 (565h57m37.535913618s) [noise=true , address=77.90.31.14:4662 , piece_id=6CJ2O7B6FHJDXHMYW7AAYB4PYLI65I3OSJLDGHNM5OAR7HMRZE2Q, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.665634514 (565h57m37.536717549s) [noise=true , address=45.91.116.12:4227 , piece_id=DJBZUKDHWYTI7GNW6LZRKOCLSVT6RN6DV57EQJBCIWLGGA6RDS4A, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:39.355263749 (565h57m39.226346784s) [upload=496206]: Next returned nil segment Dec 29 05:26:39.355284780 (565h57m39.226367815s) [upload=496206]: Waiting for error group managing segments... Dec 29 05:26:39.634829933 (565h57m39.505912978s) [noise=true , address=145.79.227.18:4007 , piece_id=SGEFJP7W6JGRKHOPS5VRKICY2EWN7E45LGG6NEYWBICF7FW37KBA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.640140114 (565h57m39.511223149s) [noise=true , address=145.79.227.15:4874 , piece_id=V2ABJ6CM3ISMEVMENDXZAY7NIEVBZFWVKN4FSNHHXY36GJEARXHA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.652374621 (565h57m39.523457667s) [noise=true , address=145.79.227.11:4729 , piece_id=KNS5P5K2TSM6LH6A574VLC74EMTJLVCITRRINSV5JSHBMVMHGL5Q, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.653433527 (565h57m39.524516572s) [noise=true , address=145.79.227.41:4087 , piece_id=UOUP4RQNZJ2SR2SZX2LKBAHZIS3PAM2DYDZ6JESKNUMBNRB3TEDA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.654073275 (565h57m39.52515632s) [noise=true , address=145.79.227.45:4231 , piece_id=7Q6ZPQP3AE35XEZS4WLDCN44FDWZ4TXQDDQ432H64VKEP4QVHETA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.660672521 (565h57m39.531755566s) [noise=true , address=145.79.227.36:4518 , piece_id=JQCTLGBLLQ5E7LVRYYUF2DCSPT52CTR6IM5NGNKSAZU2NGD4N7JA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.888353768 (565h57m39.759436803s) [noise=true , address=45.91.116.9:4261 , piece_id=HU4R532QDQTYGUVKGXV5RERNQ2Y565J7IZBXNPT4O3QLK7GDQVVA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.946926148 (565h57m39.818009183s) [noise=true , address=77.90.31.19:4842 , piece_id=EQONRO6RLYNKQFEEFIIMOCMGEVW34KORU5FW2PJ2JN44C3DRLE5A, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.950877241 (565h57m39.821960276s) [noise=true , address=45.91.116.14:4190 , piece_id=GN26T6PWDQFMD7E5SGQO4RP7IJYBJ5CPHWJZUDAHPHAOIBJYMN2A, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.968708144 (565h57m39.839791189s) [noise=true , address=45.91.116.12:4227 , piece_id=DJBZUKDHWYTI7GNW6LZRKOCLSVT6RN6DV57EQJBCIWLGGA6RDS4A, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.973452167 (565h57m39.844535212s) [noise=true , address=77.90.31.14:4662 , piece_id=6CJ2O7B6FHJDXHMYW7AAYB4PYLI65I3OSJLDGHNM5OAR7HMRZE2Q, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.975978437 (565h57m39.847061482s) [noise=true , address=77.90.31.11:4554 , piece_id=ML7UIMM4HDYWUWUW7KNNAADGLWMVBWOHO32IONXOVJBJJL55ZUEA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:40.677883801 (565h57m40.548966837s) [noise=true , address=45.91.116.6:4973 , piece_id=GMT2WKZNH6YZSG5VJQO4CEQUYQO4QEXLPDYPFFLJDE4HCXY37KZA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:40.677903079 (565h57m40.548986114s) [seg_pos={0 0}, upload=496206]: Segment reached optimal threshold of 13 pieces. Dec 29 05:26:40.678088622 (565h57m40.549171667s) [noise=true , address=77.90.31.21:4877 , piece_id=KRXZW7YVA4HX7LDJQA3YSMU6SX5H2XKKYJDI3JIZYNL5BMN367SQ, seg_pos={0 0}, upload=496206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 29 05:26:40.678101797 (565h57m40.549184832s) [noise=true , address=77.90.31.21:4877 , piece_id=KRXZW7YVA4HX7LDJQA3YSMU6SX5H2XKKYJDI3JIZYNL5BMN367SQ, seg_pos={0 0}, upload=496206]: Long tail cancellation due to optimal threshold Dec 29 05:26:40.678138607 (565h57m40.549221652s) [noise=true , address=45.91.116.7:4010 , piece_id=4DIMMGXSMZNYLKV7TCBD34YSPWIWE6EMMJTZIKWALDJH5VS4PUWA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 05:26:40.678146763 (565h57m40.549229798s) [noise=true , address=45.91.116.7:4010 , piece_id=4DIMMGXSMZNYLKV7TCBD34YSPWIWE6EMMJTZIKWALDJH5VS4PUWA, seg_pos={0 0}, upload=496206]: Long tail cancellation due to optimal threshold Dec 29 05:26:40.678251773 (565h57m40.549334828s) [noise=true , address=45.91.116.10:4083 , piece_id=LROVVOUI4LK63HJ7UKVKUNSQV6CISQKZA4DWCTHDLFCFUH2GGGHA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 29 05:26:40.678260108 (565h57m40.549343143s) [noise=true , address=45.91.116.10:4083 , piece_id=LROVVOUI4LK63HJ7UKVKUNSQV6CISQKZA4DWCTHDLFCFUH2GGGHA, seg_pos={0 0}, upload=496206]: Long tail cancellation due to optimal threshold Dec 29 05:26:40.678287440 (565h57m40.549370476s) [seg_pos={0 0}, upload=496206]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:26:40.678354178 (565h57m40.549437223s) [upload=496206]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:26:40.678359037 (565h57m40.549442092s) [upload=496206]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:26:40.980764430 (565h57m40.851847465s) [upload=496206]: Done uploading segments... Dec 29 05:26:40.980788265 (565h57m40.8518713s) [upload=496206]: Upload finished. err: <nil> Duration: 5.697296123s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................X.....X..X...X............] duration: 15.383533243s trace: https://jaeger.dev.storj.io/trace/37d27399c8bd577a FAIL (2025-12-29 05:27:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 05:18:03.821720823 (565h49m3.692803878s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822185477 (565h49m3.693268522s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822572013 (565h49m3.693655058s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823071774 (565h49m3.69415482s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823476305 (565h49m3.69455936s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823922204 (565h49m3.695005249s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.824347103 (565h49m3.695430148s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.826529919 (565h49m3.697612964s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.828201432 (565h49m3.699284477s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.829987674 (565h49m3.701070719s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.831682460 (565h49m3.702765505s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.833476788 (565h49m3.704559833s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.834480228 (565h49m3.705563274s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835861419 (565h49m3.706944454s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835997778 (565h49m3.707080823s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.837791423 (565h49m3.708874468s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:05.122896476 (565h49m4.993979511s) [upload=496083]: Next returned nil segment Dec 29 05:18:05.122911676 (565h49m4.993994721s) [upload=496083]: Waiting for error group managing segments... Dec 29 05:18:05.476685321 (565h49m5.347768356s) [noise=true , address=64.49.13.26:4946 , piece_id=SLV6JONCSMPVLEW7DM7DVVOLOOSTYE2QEQWXMX2CNLA3VRNQNEZA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.686793637 (565h49m5.557876672s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.695095065 (565h49m5.56617812s) [noise=true , address=145.79.227.3:4162 , piece_id=HCXS3I6KZSTX6EAZSOBDEMVQXX7H3ML6JGNP76Z7XUOALPPCHCOQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.879055562 (565h49m5.750138627s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.952203586 (565h49m5.823286631s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.963348518 (565h49m5.834431563s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.978376062 (565h49m5.849459097s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.029930970 (565h49m5.901014005s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.064081227 (565h49m5.935164272s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.531723043 (565h49m6.402806089s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.534889613 (565h49m6.405972658s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.634538078 (565h49m6.505621123s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.635018352 (565h49m6.506101387s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664932944 (565h49m6.53601599s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664984803 (565h49m6.536067848s) [seg_pos={0 0}, upload=496083]: Segment reached optimal threshold of 14 pieces. Dec 29 05:18:06.665226433 (565h49m6.536309478s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665236311 (565h49m6.536319366s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665410774 (565h49m6.536493819s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GHRhMSQiYGrk84jej1nYkPf1A9dxxP36cPc6suGHuVALqXF4r): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665431724 (565h49m6.536514779s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665544699 (565h49m6.536627744s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection Dec 29 05:18:06.665553366 (565h49m6.536636411s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665695977 (565h49m6.536779033s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665707088 (565h49m6.536790153s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665738198 (565h49m6.536821243s) [seg_pos={0 0}, upload=496083]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:18:06.665904214 (565h49m6.536987259s) [upload=496083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:18:06.665927027 (565h49m6.537010072s) [upload=496083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:18:06.966199067 (565h49m6.837282102s) [upload=496083]: Done uploading segments... Dec 29 05:18:06.966223153 (565h49m6.837306188s) [upload=496083]: Upload finished. err: <nil> Duration: 15.383558001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XXX........XX......X...X..XXX..X............X..X.X...XX...X] duration: 30.003902086s trace: https://jaeger.dev.storj.io/trace/741e8c2408b5518b FAIL (2025-12-29 05:27:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: DJBZUKDHWYTI7GNW6LZRKOCLSVT6RN6DV57EQJBCIWLGGA6RDS4A) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: EQONRO6RLYNKQFEEFIIMOCMGEVW34KORU5FW2PJ2JN44C3DRLE5A) context deadline exceeded ======LOGS====== Dec 29 05:26:37.406535872 (565h57m37.277618907s) [upload=496206]: Starting upload Dec 29 05:26:37.406694414 (565h57m37.277777459s) [upload=496206]: Done starting upload Dec 29 05:26:37.406964789 (565h57m37.278047824s) [upload=496206]: Uploading segments... Dec 29 05:26:37.407065891 (565h57m37.278148936s) [upload=496206]: Got next segment. Inline: false Dec 29 05:26:37.407106669 (565h57m37.278189704s) [upload=496206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:26:37.407113131 (565h57m37.278196176s) [upload=496206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:26:37.652478764 (565h57m37.523561829s) [seg_pos={0 0}, upload=496206]: Begin upload segment... Dec 29 05:26:37.652535191 (565h57m37.523618236s) [seg_pos={0 0}, upload=496206]: Done begin upload segment. Dec 29 05:26:37.652566821 (565h57m37.523649866s) [noise=true , address=77.90.31.21:4877 , piece_id=KRXZW7YVA4HX7LDJQA3YSMU6SX5H2XKKYJDI3JIZYNL5BMN367SQ, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.652597229 (565h57m37.523680264s) [noise=true , address=145.79.227.15:4874 , piece_id=V2ABJ6CM3ISMEVMENDXZAY7NIEVBZFWVKN4FSNHHXY36GJEARXHA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.653074829 (565h57m37.524157874s) [noise=true , address=77.90.31.11:4554 , piece_id=ML7UIMM4HDYWUWUW7KNNAADGLWMVBWOHO32IONXOVJBJJL55ZUEA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.653468719 (565h57m37.524551764s) [noise=true , address=77.90.31.19:4842 , piece_id=EQONRO6RLYNKQFEEFIIMOCMGEVW34KORU5FW2PJ2JN44C3DRLE5A, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.653909428 (565h57m37.524992473s) [noise=true , address=45.91.116.14:4190 , piece_id=GN26T6PWDQFMD7E5SGQO4RP7IJYBJ5CPHWJZUDAHPHAOIBJYMN2A, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.654336371 (565h57m37.525419406s) [noise=true , address=145.79.227.41:4087 , piece_id=UOUP4RQNZJ2SR2SZX2LKBAHZIS3PAM2DYDZ6JESKNUMBNRB3TEDA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.654745650 (565h57m37.525828685s) [noise=true , address=145.79.227.36:4518 , piece_id=JQCTLGBLLQ5E7LVRYYUF2DCSPT52CTR6IM5NGNKSAZU2NGD4N7JA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.655196860 (565h57m37.526279905s) [noise=true , address=145.79.227.45:4231 , piece_id=7Q6ZPQP3AE35XEZS4WLDCN44FDWZ4TXQDDQ432H64VKEP4QVHETA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.655732279 (565h57m37.526815324s) [noise=true , address=45.91.116.10:4083 , piece_id=LROVVOUI4LK63HJ7UKVKUNSQV6CISQKZA4DWCTHDLFCFUH2GGGHA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.656260625 (565h57m37.52734367s) [noise=true , address=45.91.116.6:4973 , piece_id=GMT2WKZNH6YZSG5VJQO4CEQUYQO4QEXLPDYPFFLJDE4HCXY37KZA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.657867896 (565h57m37.528950941s) [noise=true , address=145.79.227.18:4007 , piece_id=SGEFJP7W6JGRKHOPS5VRKICY2EWN7E45LGG6NEYWBICF7FW37KBA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.659488141 (565h57m37.530571186s) [noise=true , address=45.91.116.9:4261 , piece_id=HU4R532QDQTYGUVKGXV5RERNQ2Y565J7IZBXNPT4O3QLK7GDQVVA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.661071716 (565h57m37.532154761s) [noise=true , address=45.91.116.7:4010 , piece_id=4DIMMGXSMZNYLKV7TCBD34YSPWIWE6EMMJTZIKWALDJH5VS4PUWA, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.662928932 (565h57m37.534011977s) [noise=true , address=145.79.227.11:4729 , piece_id=KNS5P5K2TSM6LH6A574VLC74EMTJLVCITRRINSV5JSHBMVMHGL5Q, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.664830573 (565h57m37.535913618s) [noise=true , address=77.90.31.14:4662 , piece_id=6CJ2O7B6FHJDXHMYW7AAYB4PYLI65I3OSJLDGHNM5OAR7HMRZE2Q, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:37.665634514 (565h57m37.536717549s) [noise=true , address=45.91.116.12:4227 , piece_id=DJBZUKDHWYTI7GNW6LZRKOCLSVT6RN6DV57EQJBCIWLGGA6RDS4A, seg_pos={0 0}, upload=496206]: Uploading piece... Dec 29 05:26:39.355263749 (565h57m39.226346784s) [upload=496206]: Next returned nil segment Dec 29 05:26:39.355284780 (565h57m39.226367815s) [upload=496206]: Waiting for error group managing segments... Dec 29 05:26:39.634829933 (565h57m39.505912978s) [noise=true , address=145.79.227.18:4007 , piece_id=SGEFJP7W6JGRKHOPS5VRKICY2EWN7E45LGG6NEYWBICF7FW37KBA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.640140114 (565h57m39.511223149s) [noise=true , address=145.79.227.15:4874 , piece_id=V2ABJ6CM3ISMEVMENDXZAY7NIEVBZFWVKN4FSNHHXY36GJEARXHA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.652374621 (565h57m39.523457667s) [noise=true , address=145.79.227.11:4729 , piece_id=KNS5P5K2TSM6LH6A574VLC74EMTJLVCITRRINSV5JSHBMVMHGL5Q, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.653433527 (565h57m39.524516572s) [noise=true , address=145.79.227.41:4087 , piece_id=UOUP4RQNZJ2SR2SZX2LKBAHZIS3PAM2DYDZ6JESKNUMBNRB3TEDA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.654073275 (565h57m39.52515632s) [noise=true , address=145.79.227.45:4231 , piece_id=7Q6ZPQP3AE35XEZS4WLDCN44FDWZ4TXQDDQ432H64VKEP4QVHETA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.660672521 (565h57m39.531755566s) [noise=true , address=145.79.227.36:4518 , piece_id=JQCTLGBLLQ5E7LVRYYUF2DCSPT52CTR6IM5NGNKSAZU2NGD4N7JA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.888353768 (565h57m39.759436803s) [noise=true , address=45.91.116.9:4261 , piece_id=HU4R532QDQTYGUVKGXV5RERNQ2Y565J7IZBXNPT4O3QLK7GDQVVA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.946926148 (565h57m39.818009183s) [noise=true , address=77.90.31.19:4842 , piece_id=EQONRO6RLYNKQFEEFIIMOCMGEVW34KORU5FW2PJ2JN44C3DRLE5A, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.950877241 (565h57m39.821960276s) [noise=true , address=45.91.116.14:4190 , piece_id=GN26T6PWDQFMD7E5SGQO4RP7IJYBJ5CPHWJZUDAHPHAOIBJYMN2A, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.968708144 (565h57m39.839791189s) [noise=true , address=45.91.116.12:4227 , piece_id=DJBZUKDHWYTI7GNW6LZRKOCLSVT6RN6DV57EQJBCIWLGGA6RDS4A, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.973452167 (565h57m39.844535212s) [noise=true , address=77.90.31.14:4662 , piece_id=6CJ2O7B6FHJDXHMYW7AAYB4PYLI65I3OSJLDGHNM5OAR7HMRZE2Q, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:39.975978437 (565h57m39.847061482s) [noise=true , address=77.90.31.11:4554 , piece_id=ML7UIMM4HDYWUWUW7KNNAADGLWMVBWOHO32IONXOVJBJJL55ZUEA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:40.677883801 (565h57m40.548966837s) [noise=true , address=45.91.116.6:4973 , piece_id=GMT2WKZNH6YZSG5VJQO4CEQUYQO4QEXLPDYPFFLJDE4HCXY37KZA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: <nil> Dec 29 05:26:40.677903079 (565h57m40.548986114s) [seg_pos={0 0}, upload=496206]: Segment reached optimal threshold of 13 pieces. Dec 29 05:26:40.678088622 (565h57m40.549171667s) [noise=true , address=77.90.31.21:4877 , piece_id=KRXZW7YVA4HX7LDJQA3YSMU6SX5H2XKKYJDI3JIZYNL5BMN367SQ, seg_pos={0 0}, upload=496206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 29 05:26:40.678101797 (565h57m40.549184832s) [noise=true , address=77.90.31.21:4877 , piece_id=KRXZW7YVA4HX7LDJQA3YSMU6SX5H2XKKYJDI3JIZYNL5BMN367SQ, seg_pos={0 0}, upload=496206]: Long tail cancellation due to optimal threshold Dec 29 05:26:40.678138607 (565h57m40.549221652s) [noise=true , address=45.91.116.7:4010 , piece_id=4DIMMGXSMZNYLKV7TCBD34YSPWIWE6EMMJTZIKWALDJH5VS4PUWA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 05:26:40.678146763 (565h57m40.549229798s) [noise=true , address=45.91.116.7:4010 , piece_id=4DIMMGXSMZNYLKV7TCBD34YSPWIWE6EMMJTZIKWALDJH5VS4PUWA, seg_pos={0 0}, upload=496206]: Long tail cancellation due to optimal threshold Dec 29 05:26:40.678251773 (565h57m40.549334828s) [noise=true , address=45.91.116.10:4083 , piece_id=LROVVOUI4LK63HJ7UKVKUNSQV6CISQKZA4DWCTHDLFCFUH2GGGHA, seg_pos={0 0}, upload=496206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 29 05:26:40.678260108 (565h57m40.549343143s) [noise=true , address=45.91.116.10:4083 , piece_id=LROVVOUI4LK63HJ7UKVKUNSQV6CISQKZA4DWCTHDLFCFUH2GGGHA, seg_pos={0 0}, upload=496206]: Long tail cancellation due to optimal threshold Dec 29 05:26:40.678287440 (565h57m40.549370476s) [seg_pos={0 0}, upload=496206]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:26:40.678354178 (565h57m40.549437223s) [upload=496206]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:26:40.678359037 (565h57m40.549442092s) [upload=496206]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:26:40.980764430 (565h57m40.851847465s) [upload=496206]: Done uploading segments... Dec 29 05:26:40.980788265 (565h57m40.8518713s) [upload=496206]: Upload finished. err: <nil> Duration: 30.003929869s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....X.....X........X..X...X....X....X.........X...X.........] duration: 3.642413884s trace: https://jaeger.dev.storj.io/trace/159c3b7a9a64e4e4 FAIL (2025-12-29 05:27:05 UTC) orbiter warning: context deadline exceeded Duration: 3.642595338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................X.....X..X...X...........] duration: 21.572174585s trace: https://jaeger.dev.storj.io/trace/37d27399c8bd577a FAIL (2025-12-29 05:26:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 05:18:03.821720823 (565h49m3.692803878s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822185477 (565h49m3.693268522s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822572013 (565h49m3.693655058s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823071774 (565h49m3.69415482s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823476305 (565h49m3.69455936s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823922204 (565h49m3.695005249s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.824347103 (565h49m3.695430148s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.826529919 (565h49m3.697612964s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.828201432 (565h49m3.699284477s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.829987674 (565h49m3.701070719s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.831682460 (565h49m3.702765505s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.833476788 (565h49m3.704559833s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.834480228 (565h49m3.705563274s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835861419 (565h49m3.706944454s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835997778 (565h49m3.707080823s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.837791423 (565h49m3.708874468s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:05.122896476 (565h49m4.993979511s) [upload=496083]: Next returned nil segment Dec 29 05:18:05.122911676 (565h49m4.993994721s) [upload=496083]: Waiting for error group managing segments... Dec 29 05:18:05.476685321 (565h49m5.347768356s) [noise=true , address=64.49.13.26:4946 , piece_id=SLV6JONCSMPVLEW7DM7DVVOLOOSTYE2QEQWXMX2CNLA3VRNQNEZA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.686793637 (565h49m5.557876672s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.695095065 (565h49m5.56617812s) [noise=true , address=145.79.227.3:4162 , piece_id=HCXS3I6KZSTX6EAZSOBDEMVQXX7H3ML6JGNP76Z7XUOALPPCHCOQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.879055562 (565h49m5.750138627s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.952203586 (565h49m5.823286631s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.963348518 (565h49m5.834431563s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.978376062 (565h49m5.849459097s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.029930970 (565h49m5.901014005s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.064081227 (565h49m5.935164272s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.531723043 (565h49m6.402806089s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.534889613 (565h49m6.405972658s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.634538078 (565h49m6.505621123s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.635018352 (565h49m6.506101387s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664932944 (565h49m6.53601599s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664984803 (565h49m6.536067848s) [seg_pos={0 0}, upload=496083]: Segment reached optimal threshold of 14 pieces. Dec 29 05:18:06.665226433 (565h49m6.536309478s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665236311 (565h49m6.536319366s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665410774 (565h49m6.536493819s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GHRhMSQiYGrk84jej1nYkPf1A9dxxP36cPc6suGHuVALqXF4r): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665431724 (565h49m6.536514779s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665544699 (565h49m6.536627744s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection Dec 29 05:18:06.665553366 (565h49m6.536636411s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665695977 (565h49m6.536779033s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665707088 (565h49m6.536790153s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665738198 (565h49m6.536821243s) [seg_pos={0 0}, upload=496083]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:18:06.665904214 (565h49m6.536987259s) [upload=496083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:18:06.665927027 (565h49m6.537010072s) [upload=496083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:18:06.966199067 (565h49m6.837282102s) [upload=496083]: Done uploading segments... Dec 29 05:18:06.966223153 (565h49m6.837306188s) [upload=496083]: Upload finished. err: <nil> Duration: 21.572200934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................X.....X..X...X..........] duration: 33.001500512s trace: https://jaeger.dev.storj.io/trace/37d27399c8bd577a FAIL (2025-12-29 05:26:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 05:18:03.821720823 (565h49m3.692803878s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822185477 (565h49m3.693268522s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822572013 (565h49m3.693655058s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823071774 (565h49m3.69415482s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823476305 (565h49m3.69455936s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823922204 (565h49m3.695005249s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.824347103 (565h49m3.695430148s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.826529919 (565h49m3.697612964s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.828201432 (565h49m3.699284477s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.829987674 (565h49m3.701070719s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.831682460 (565h49m3.702765505s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.833476788 (565h49m3.704559833s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.834480228 (565h49m3.705563274s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835861419 (565h49m3.706944454s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835997778 (565h49m3.707080823s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.837791423 (565h49m3.708874468s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:05.122896476 (565h49m4.993979511s) [upload=496083]: Next returned nil segment Dec 29 05:18:05.122911676 (565h49m4.993994721s) [upload=496083]: Waiting for error group managing segments... Dec 29 05:18:05.476685321 (565h49m5.347768356s) [noise=true , address=64.49.13.26:4946 , piece_id=SLV6JONCSMPVLEW7DM7DVVOLOOSTYE2QEQWXMX2CNLA3VRNQNEZA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.686793637 (565h49m5.557876672s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.695095065 (565h49m5.56617812s) [noise=true , address=145.79.227.3:4162 , piece_id=HCXS3I6KZSTX6EAZSOBDEMVQXX7H3ML6JGNP76Z7XUOALPPCHCOQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.879055562 (565h49m5.750138627s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.952203586 (565h49m5.823286631s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.963348518 (565h49m5.834431563s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.978376062 (565h49m5.849459097s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.029930970 (565h49m5.901014005s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.064081227 (565h49m5.935164272s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.531723043 (565h49m6.402806089s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.534889613 (565h49m6.405972658s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.634538078 (565h49m6.505621123s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.635018352 (565h49m6.506101387s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664932944 (565h49m6.53601599s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664984803 (565h49m6.536067848s) [seg_pos={0 0}, upload=496083]: Segment reached optimal threshold of 14 pieces. Dec 29 05:18:06.665226433 (565h49m6.536309478s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665236311 (565h49m6.536319366s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665410774 (565h49m6.536493819s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GHRhMSQiYGrk84jej1nYkPf1A9dxxP36cPc6suGHuVALqXF4r): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665431724 (565h49m6.536514779s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665544699 (565h49m6.536627744s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection Dec 29 05:18:06.665553366 (565h49m6.536636411s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665695977 (565h49m6.536779033s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665707088 (565h49m6.536790153s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665738198 (565h49m6.536821243s) [seg_pos={0 0}, upload=496083]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:18:06.665904214 (565h49m6.536987259s) [upload=496083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:18:06.665927027 (565h49m6.537010072s) [upload=496083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:18:06.966199067 (565h49m6.837282102s) [upload=496083]: Done uploading segments... Dec 29 05:18:06.966223153 (565h49m6.837306188s) [upload=496083]: Upload finished. err: <nil> Duration: 33.001514289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..XXX........XX......X...X..XXX..X............X..X.X...XX...] duration: 6.211370002s trace: https://jaeger.dev.storj.io/trace/291592605b9f77be FAIL (2025-12-29 05:26:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 05:24:47.259947029 (565h55m47.131030064s) [upload=496177]: Starting upload Dec 29 05:24:47.260005089 (565h55m47.131088134s) [upload=496177]: Done starting upload Dec 29 05:24:47.260167499 (565h55m47.131250544s) [upload=496177]: Uploading segments... Dec 29 05:24:47.260245487 (565h55m47.131328522s) [upload=496177]: Got next segment. Inline: false Dec 29 05:24:47.260308928 (565h55m47.131391973s) [upload=496177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:24:47.260316692 (565h55m47.131399728s) [upload=496177]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:24:47.506070944 (565h55m47.377153979s) [seg_pos={0 0}, upload=496177]: Begin upload segment... Dec 29 05:24:47.506119246 (565h55m47.377202281s) [seg_pos={0 0}, upload=496177]: Done begin upload segment. Dec 29 05:24:47.506160334 (565h55m47.377243379s) [noise=true , address=77.90.31.13:4627 , piece_id=MJ57NKNPOZBCRGV47V2O6MZ5AMUC7RUWOXHTE7YAW37OI6Q3UKNA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.506280624 (565h55m47.377363659s) [noise=true , address=145.79.227.45:4230 , piece_id=A4RFQL2LD42H2VNKH533LDSE2PEWYJNS6T3QZTDM6JL2Q4MWQ7EA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.506874013 (565h55m47.377957048s) [noise=true , address=145.79.227.6:4257 , piece_id=KH7JYCCPG7V7GFPSY7NX4K4GG7M2VV3JYVVOD7N2HI6YWDAHG4CA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.507123749 (565h55m47.378206794s) [noise=true , address=77.90.31.10:4519 , piece_id=IY2ADV6FEPZS4DUJ4NUR4NBKXCCOLKBI5X6UMY6UKM53FNYP56EA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.507524743 (565h55m47.378607788s) [noise=true , address=145.79.227.46:4603 , piece_id=2SSW7O2HHHV5KBGBNMAPFD2HCLZZDTMPTR3ENDLPVQD53GQ6VT3A, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.507902051 (565h55m47.378985096s) [noise=true , address=45.91.116.7:4011 , piece_id=XQYPLTH3DKGGTJC34E4KAD2G2MHKZ4UZGREQDF5WOBRZXKE6MBCQ, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.508284038 (565h55m47.379367083s) [noise=true , address=145.79.227.48:4303 , piece_id=H6JVKF76M4F24BRG73FSNKSESVZABZ27FA6ENLGPVSV4DBI3FHYQ, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.508670304 (565h55m47.379753339s) [noise=true , address=77.90.31.19:4842 , piece_id=KJ5EB6QDRPV7ZAEJMYH3SPG6WCL6HYZFYRQWKGRR5N55Q7UOD5LQ, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.509077389 (565h55m47.380160424s) [noise=true , address=145.79.227.15:4873 , piece_id=WJBG6SGY4GEYRBWPCC5AG6MGEUSOL2K7TB4V3JIQ5D6ZQ2EJ3YHA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.509520863 (565h55m47.380603908s) [noise=true , address=45.91.116.4:4939 , piece_id=YNKOKBCJSHNH2VCG3GS4XEAJK2W4VJG2RWLIZORPHHJMVVNASJGA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.509980498 (565h55m47.381063544s) [noise=true , address=145.79.227.42:4122 , piece_id=SLI6VYEI2DIFCDC4LVDHXTWPSBQ536RTZDVXQOTW6G6ZCJ2ENL3Q, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.510107250 (565h55m47.381190285s) [noise=true , address=77.90.31.16:4735 , piece_id=6QBDKLH4Q44DS7GHFT47KTPWKVBQDBPA2RJING4UGTJS6H5J4KEA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.510409185 (565h55m47.38149223s) [noise=true , address=145.79.227.41:4087 , piece_id=CN56ZHZNNIACHGPTPVTILJOIQBJRHMSNU34FVI5MNTGGELLN75AQ, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.510423222 (565h55m47.381506257s) [noise=true , address=45.91.116.14:4189 , piece_id=W64YUT6AQ5STTW5LX7VJDCGXSUA62VYFSGSPFOXKRIFNWMUGULPA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.510922733 (565h55m47.382005768s) [noise=true , address=145.79.227.21:4473 , piece_id=ZF4PYDJNAYOADT3CEANDQX6PMRKU3ZLKC6IPBZWOOOFMIAHFJ5BA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.511310150 (565h55m47.382393185s) [noise=true , address=145.79.227.26:4654 , piece_id=I7UEZZGDPNASYHWIBVNEXPBXAW55UE4JDNDK54XIU7I7OPZY2CRA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:49.197215253 (565h55m49.068298298s) [upload=496177]: Next returned nil segment Dec 29 05:24:49.197231054 (565h55m49.068314099s) [upload=496177]: Waiting for error group managing segments... Dec 29 05:24:49.478413174 (565h55m49.34949622s) [noise=true , address=145.79.227.45:4230 , piece_id=A4RFQL2LD42H2VNKH533LDSE2PEWYJNS6T3QZTDM6JL2Q4MWQ7EA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.498156930 (565h55m49.369239975s) [noise=true , address=145.79.227.26:4654 , piece_id=I7UEZZGDPNASYHWIBVNEXPBXAW55UE4JDNDK54XIU7I7OPZY2CRA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.499262935 (565h55m49.37034597s) [noise=true , address=145.79.227.46:4603 , piece_id=2SSW7O2HHHV5KBGBNMAPFD2HCLZZDTMPTR3ENDLPVQD53GQ6VT3A, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.501031092 (565h55m49.372114127s) [noise=true , address=145.79.227.41:4087 , piece_id=CN56ZHZNNIACHGPTPVTILJOIQBJRHMSNU34FVI5MNTGGELLN75AQ, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.517640107 (565h55m49.388723143s) [noise=true , address=145.79.227.48:4303 , piece_id=H6JVKF76M4F24BRG73FSNKSESVZABZ27FA6ENLGPVSV4DBI3FHYQ, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.518412017 (565h55m49.389495052s) [noise=true , address=145.79.227.15:4873 , piece_id=WJBG6SGY4GEYRBWPCC5AG6MGEUSOL2K7TB4V3JIQ5D6ZQ2EJ3YHA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.527292317 (565h55m49.398375362s) [noise=true , address=145.79.227.6:4257 , piece_id=KH7JYCCPG7V7GFPSY7NX4K4GG7M2VV3JYVVOD7N2HI6YWDAHG4CA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.538294827 (565h55m49.409377862s) [noise=true , address=145.79.227.42:4122 , piece_id=SLI6VYEI2DIFCDC4LVDHXTWPSBQ536RTZDVXQOTW6G6ZCJ2ENL3Q, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.642293897 (565h55m49.513376932s) [noise=true , address=145.79.227.21:4473 , piece_id=ZF4PYDJNAYOADT3CEANDQX6PMRKU3ZLKC6IPBZWOOOFMIAHFJ5BA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.674396034 (565h55m49.545479079s) [noise=true , address=77.90.31.19:4842 , piece_id=KJ5EB6QDRPV7ZAEJMYH3SPG6WCL6HYZFYRQWKGRR5N55Q7UOD5LQ, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.694408821 (565h55m49.565491856s) [noise=true , address=77.90.31.13:4627 , piece_id=MJ57NKNPOZBCRGV47V2O6MZ5AMUC7RUWOXHTE7YAW37OI6Q3UKNA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.947202917 (565h55m49.818285952s) [noise=true , address=45.91.116.14:4189 , piece_id=W64YUT6AQ5STTW5LX7VJDCGXSUA62VYFSGSPFOXKRIFNWMUGULPA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:50.034551511 (565h55m49.905634546s) [noise=true , address=77.90.31.10:4519 , piece_id=IY2ADV6FEPZS4DUJ4NUR4NBKXCCOLKBI5X6UMY6UKM53FNYP56EA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:50.034569975 (565h55m49.905653011s) [seg_pos={0 0}, upload=496177]: Segment reached optimal threshold of 13 pieces. Dec 29 05:24:50.034785406 (565h55m49.905868451s) [noise=true , address=77.90.31.16:4735 , piece_id=6QBDKLH4Q44DS7GHFT47KTPWKVBQDBPA2RJING4UGTJS6H5J4KEA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 29 05:24:50.034792780 (565h55m49.905875815s) [noise=true , address=77.90.31.16:4735 , piece_id=6QBDKLH4Q44DS7GHFT47KTPWKVBQDBPA2RJING4UGTJS6H5J4KEA, seg_pos={0 0}, upload=496177]: Long tail cancellation due to optimal threshold Dec 29 05:24:50.034914672 (565h55m49.905997717s) [noise=true , address=45.91.116.4:4939 , piece_id=YNKOKBCJSHNH2VCG3GS4XEAJK2W4VJG2RWLIZORPHHJMVVNASJGA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 05:24:50.034920803 (565h55m49.906003838s) [noise=true , address=45.91.116.4:4939 , piece_id=YNKOKBCJSHNH2VCG3GS4XEAJK2W4VJG2RWLIZORPHHJMVVNASJGA, seg_pos={0 0}, upload=496177]: Long tail cancellation due to optimal threshold Dec 29 05:24:50.035124131 (565h55m49.906207176s) [noise=true , address=45.91.116.7:4011 , piece_id=XQYPLTH3DKGGTJC34E4KAD2G2MHKZ4UZGREQDF5WOBRZXKE6MBCQ, seg_pos={0 0}, upload=496177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:49596->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:49596->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49596->45.91.116.7:4011: use of closed network connection Dec 29 05:24:50.035139350 (565h55m49.906222395s) [noise=true , address=45.91.116.7:4011 , piece_id=XQYPLTH3DKGGTJC34E4KAD2G2MHKZ4UZGREQDF5WOBRZXKE6MBCQ, seg_pos={0 0}, upload=496177]: Long tail cancellation due to optimal threshold Dec 29 05:24:50.035162253 (565h55m49.906245298s) [seg_pos={0 0}, upload=496177]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:24:50.035254890 (565h55m49.906337935s) [upload=496177]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:24:50.035260270 (565h55m49.906343305s) [upload=496177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:24:50.332008061 (565h55m50.203091126s) [upload=496177]: Done uploading segments... Dec 29 05:24:50.332032207 (565h55m50.203115252s) [upload=496177]: Upload finished. err: <nil> Duration: 6.211396472s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.....X.....X........X..X...X....X....X.........X...X........] duration: 4.282559275s trace: https://jaeger.dev.storj.io/trace/159c3b7a9a64e4e4 FAIL (2025-12-29 05:26:10 UTC) orbiter warning: context deadline exceeded Duration: 4.28257849s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................X.....X..X...X.........] duration: 51.91325983s trace: https://jaeger.dev.storj.io/trace/37d27399c8bd577a FAIL (2025-12-29 05:25:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 05:18:03.821720823 (565h49m3.692803878s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822185477 (565h49m3.693268522s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.822572013 (565h49m3.693655058s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823071774 (565h49m3.69415482s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823476305 (565h49m3.69455936s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.823922204 (565h49m3.695005249s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.824347103 (565h49m3.695430148s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.826529919 (565h49m3.697612964s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.828201432 (565h49m3.699284477s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.829987674 (565h49m3.701070719s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.831682460 (565h49m3.702765505s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.833476788 (565h49m3.704559833s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.834480228 (565h49m3.705563274s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835861419 (565h49m3.706944454s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.835997778 (565h49m3.707080823s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:03.837791423 (565h49m3.708874468s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Uploading piece... Dec 29 05:18:05.122896476 (565h49m4.993979511s) [upload=496083]: Next returned nil segment Dec 29 05:18:05.122911676 (565h49m4.993994721s) [upload=496083]: Waiting for error group managing segments... Dec 29 05:18:05.476685321 (565h49m5.347768356s) [noise=true , address=64.49.13.26:4946 , piece_id=SLV6JONCSMPVLEW7DM7DVVOLOOSTYE2QEQWXMX2CNLA3VRNQNEZA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.686793637 (565h49m5.557876672s) [noise=true , address=145.79.227.14:4858 , piece_id=7UCC6MIA4AZKVQL5GKF6UPYU2SWE5LVJTPKAXULZBS3EYK6WEXYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.695095065 (565h49m5.56617812s) [noise=true , address=145.79.227.3:4162 , piece_id=HCXS3I6KZSTX6EAZSOBDEMVQXX7H3ML6JGNP76Z7XUOALPPCHCOQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.879055562 (565h49m5.750138627s) [noise=true , address=77.90.31.16:4768 , piece_id=NJH2FOLRIQVMUKBQKZBRN3PLDX5CJYWBGHRORYR6NY6OTOYCMEMA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.952203586 (565h49m5.823286631s) [noise=true , address=77.90.31.15:4712 , piece_id=7TKBK746K6ZJBZVGGKC26VG35G2ZBNHVYUVI6XMOBRBCIKCMTE4Q, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.963348518 (565h49m5.834431563s) [noise=true , address=77.90.31.14:4673 , piece_id=V6HREI6HMSFVKIH4JMDBHPCUUMIJY7O4WPYKUO3TJIEFU26SQZGA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:05.978376062 (565h49m5.849459097s) [noise=true , address=77.90.31.13:4638 , piece_id=575WCAWQ5FOU64JMSDEBF6BMRNKMPZ44JXFWKXX5I4R4BYPRHOHQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.029930970 (565h49m5.901014005s) [noise=true , address=45.91.116.14:4207 , piece_id=AGGN3QUTYB4O6OSRDVH2B7NAWBIEKRW4BJB3CPCPYN4Z22GHPWPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.064081227 (565h49m5.935164272s) [noise=true , address=45.91.116.14:4221 , piece_id=NLESLTLSMUZWQPXD4NQKPM5VIMZ2YNSMCCF35TFRYRLN5ZVJ3EPQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.531723043 (565h49m6.402806089s) [noise=true , address=145.79.227.30:4781 , piece_id=ZRSAJYBBSDOBJ5PQOUZQX6JN6TX5ZXJCYUVNBLM3YRGLO4XO7NYA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.534889613 (565h49m6.405972658s) [noise=true , address=77.90.31.11:4572 , piece_id=HV65W56BH5GORX2CIQEV32YUS7RDRXNUT6QHGLJAPZDGBJXUXSWQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.634538078 (565h49m6.505621123s) [noise=true , address=145.79.227.33:4880 , piece_id=LW36YV6WXM6IFCFCD6UL2BRF6ND5H2DD5DZGPE5YVYYKC7L3FZCQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.635018352 (565h49m6.506101387s) [noise=true , address=145.79.227.34:4935 , piece_id=YBI5MJI5X5O7TTE7QEU2Z4OAAPZ7TTLPDNGD7H7MZOCA4FH4KZCA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664932944 (565h49m6.53601599s) [noise=true , address=145.79.227.40:4055 , piece_id=H7MHNFCPALHAPEY5VHMTQQS7NIMQ6OOY3VY4FN3I7Z2HGHYM2LHA, seg_pos={0 0}, upload=496083]: Done uploading piece. err: <nil> Dec 29 05:18:06.664984803 (565h49m6.536067848s) [seg_pos={0 0}, upload=496083]: Segment reached optimal threshold of 14 pieces. Dec 29 05:18:06.665226433 (565h49m6.536309478s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665236311 (565h49m6.536319366s) [noise=true , address=193.149.164.143:4694 , piece_id=ER5QDRZVIANPP7R7YFVQFOZE5H34T6O25CBH2ZMSSZIEPDGIUONQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665410774 (565h49m6.536493819s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GHRhMSQiYGrk84jej1nYkPf1A9dxxP36cPc6suGHuVALqXF4r): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665431724 (565h49m6.536514779s) [noise=true , address=45.91.116.8:4075 , piece_id=RJ6H4VWLS46SBPCRTVMYSXDR24JH67CKKKPCIWOUFGLO7267XXUQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665544699 (565h49m6.536627744s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->64.49.13.169:4706: use of closed network connection Dec 29 05:18:06.665553366 (565h49m6.536636411s) [noise=true , address=64.49.13.169:4706 , piece_id=4WMMIUPH4XJCSZNPUWI5LVD22DKFRM732XW2C2EVKO5MVOMCREGQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665695977 (565h49m6.536779033s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:18:06.665707088 (565h49m6.536790153s) [noise=true , address=145.79.227.43:4184 , piece_id=ANPONHULUFNISTX6J4XX63YBVFQBI57KEA7W53IR7CKMMDHPWZBQ, seg_pos={0 0}, upload=496083]: Long tail cancellation due to optimal threshold Dec 29 05:18:06.665738198 (565h49m6.536821243s) [seg_pos={0 0}, upload=496083]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:18:06.665904214 (565h49m6.536987259s) [upload=496083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:18:06.665927027 (565h49m6.537010072s) [upload=496083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:18:06.966199067 (565h49m6.837282102s) [upload=496083]: Done uploading segments... Dec 29 05:18:06.966223153 (565h49m6.837306188s) [upload=496083]: Upload finished. err: <nil> Duration: 51.913282712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X..XXX........XX......X...X..XXX..X............X..X.X...XX..] duration: 5.643135518s trace: https://jaeger.dev.storj.io/trace/291592605b9f77be FAIL (2025-12-29 05:25:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 05:24:47.259947029 (565h55m47.131030064s) [upload=496177]: Starting upload Dec 29 05:24:47.260005089 (565h55m47.131088134s) [upload=496177]: Done starting upload Dec 29 05:24:47.260167499 (565h55m47.131250544s) [upload=496177]: Uploading segments... Dec 29 05:24:47.260245487 (565h55m47.131328522s) [upload=496177]: Got next segment. Inline: false Dec 29 05:24:47.260308928 (565h55m47.131391973s) [upload=496177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:24:47.260316692 (565h55m47.131399728s) [upload=496177]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:24:47.506070944 (565h55m47.377153979s) [seg_pos={0 0}, upload=496177]: Begin upload segment... Dec 29 05:24:47.506119246 (565h55m47.377202281s) [seg_pos={0 0}, upload=496177]: Done begin upload segment. Dec 29 05:24:47.506160334 (565h55m47.377243379s) [noise=true , address=77.90.31.13:4627 , piece_id=MJ57NKNPOZBCRGV47V2O6MZ5AMUC7RUWOXHTE7YAW37OI6Q3UKNA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.506280624 (565h55m47.377363659s) [noise=true , address=145.79.227.45:4230 , piece_id=A4RFQL2LD42H2VNKH533LDSE2PEWYJNS6T3QZTDM6JL2Q4MWQ7EA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.506874013 (565h55m47.377957048s) [noise=true , address=145.79.227.6:4257 , piece_id=KH7JYCCPG7V7GFPSY7NX4K4GG7M2VV3JYVVOD7N2HI6YWDAHG4CA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.507123749 (565h55m47.378206794s) [noise=true , address=77.90.31.10:4519 , piece_id=IY2ADV6FEPZS4DUJ4NUR4NBKXCCOLKBI5X6UMY6UKM53FNYP56EA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.507524743 (565h55m47.378607788s) [noise=true , address=145.79.227.46:4603 , piece_id=2SSW7O2HHHV5KBGBNMAPFD2HCLZZDTMPTR3ENDLPVQD53GQ6VT3A, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.507902051 (565h55m47.378985096s) [noise=true , address=45.91.116.7:4011 , piece_id=XQYPLTH3DKGGTJC34E4KAD2G2MHKZ4UZGREQDF5WOBRZXKE6MBCQ, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.508284038 (565h55m47.379367083s) [noise=true , address=145.79.227.48:4303 , piece_id=H6JVKF76M4F24BRG73FSNKSESVZABZ27FA6ENLGPVSV4DBI3FHYQ, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.508670304 (565h55m47.379753339s) [noise=true , address=77.90.31.19:4842 , piece_id=KJ5EB6QDRPV7ZAEJMYH3SPG6WCL6HYZFYRQWKGRR5N55Q7UOD5LQ, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.509077389 (565h55m47.380160424s) [noise=true , address=145.79.227.15:4873 , piece_id=WJBG6SGY4GEYRBWPCC5AG6MGEUSOL2K7TB4V3JIQ5D6ZQ2EJ3YHA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.509520863 (565h55m47.380603908s) [noise=true , address=45.91.116.4:4939 , piece_id=YNKOKBCJSHNH2VCG3GS4XEAJK2W4VJG2RWLIZORPHHJMVVNASJGA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.509980498 (565h55m47.381063544s) [noise=true , address=145.79.227.42:4122 , piece_id=SLI6VYEI2DIFCDC4LVDHXTWPSBQ536RTZDVXQOTW6G6ZCJ2ENL3Q, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.510107250 (565h55m47.381190285s) [noise=true , address=77.90.31.16:4735 , piece_id=6QBDKLH4Q44DS7GHFT47KTPWKVBQDBPA2RJING4UGTJS6H5J4KEA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.510409185 (565h55m47.38149223s) [noise=true , address=145.79.227.41:4087 , piece_id=CN56ZHZNNIACHGPTPVTILJOIQBJRHMSNU34FVI5MNTGGELLN75AQ, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.510423222 (565h55m47.381506257s) [noise=true , address=45.91.116.14:4189 , piece_id=W64YUT6AQ5STTW5LX7VJDCGXSUA62VYFSGSPFOXKRIFNWMUGULPA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.510922733 (565h55m47.382005768s) [noise=true , address=145.79.227.21:4473 , piece_id=ZF4PYDJNAYOADT3CEANDQX6PMRKU3ZLKC6IPBZWOOOFMIAHFJ5BA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.511310150 (565h55m47.382393185s) [noise=true , address=145.79.227.26:4654 , piece_id=I7UEZZGDPNASYHWIBVNEXPBXAW55UE4JDNDK54XIU7I7OPZY2CRA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:49.197215253 (565h55m49.068298298s) [upload=496177]: Next returned nil segment Dec 29 05:24:49.197231054 (565h55m49.068314099s) [upload=496177]: Waiting for error group managing segments... Dec 29 05:24:49.478413174 (565h55m49.34949622s) [noise=true , address=145.79.227.45:4230 , piece_id=A4RFQL2LD42H2VNKH533LDSE2PEWYJNS6T3QZTDM6JL2Q4MWQ7EA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.498156930 (565h55m49.369239975s) [noise=true , address=145.79.227.26:4654 , piece_id=I7UEZZGDPNASYHWIBVNEXPBXAW55UE4JDNDK54XIU7I7OPZY2CRA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.499262935 (565h55m49.37034597s) [noise=true , address=145.79.227.46:4603 , piece_id=2SSW7O2HHHV5KBGBNMAPFD2HCLZZDTMPTR3ENDLPVQD53GQ6VT3A, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.501031092 (565h55m49.372114127s) [noise=true , address=145.79.227.41:4087 , piece_id=CN56ZHZNNIACHGPTPVTILJOIQBJRHMSNU34FVI5MNTGGELLN75AQ, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.517640107 (565h55m49.388723143s) [noise=true , address=145.79.227.48:4303 , piece_id=H6JVKF76M4F24BRG73FSNKSESVZABZ27FA6ENLGPVSV4DBI3FHYQ, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.518412017 (565h55m49.389495052s) [noise=true , address=145.79.227.15:4873 , piece_id=WJBG6SGY4GEYRBWPCC5AG6MGEUSOL2K7TB4V3JIQ5D6ZQ2EJ3YHA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.527292317 (565h55m49.398375362s) [noise=true , address=145.79.227.6:4257 , piece_id=KH7JYCCPG7V7GFPSY7NX4K4GG7M2VV3JYVVOD7N2HI6YWDAHG4CA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.538294827 (565h55m49.409377862s) [noise=true , address=145.79.227.42:4122 , piece_id=SLI6VYEI2DIFCDC4LVDHXTWPSBQ536RTZDVXQOTW6G6ZCJ2ENL3Q, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.642293897 (565h55m49.513376932s) [noise=true , address=145.79.227.21:4473 , piece_id=ZF4PYDJNAYOADT3CEANDQX6PMRKU3ZLKC6IPBZWOOOFMIAHFJ5BA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.674396034 (565h55m49.545479079s) [noise=true , address=77.90.31.19:4842 , piece_id=KJ5EB6QDRPV7ZAEJMYH3SPG6WCL6HYZFYRQWKGRR5N55Q7UOD5LQ, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.694408821 (565h55m49.565491856s) [noise=true , address=77.90.31.13:4627 , piece_id=MJ57NKNPOZBCRGV47V2O6MZ5AMUC7RUWOXHTE7YAW37OI6Q3UKNA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.947202917 (565h55m49.818285952s) [noise=true , address=45.91.116.14:4189 , piece_id=W64YUT6AQ5STTW5LX7VJDCGXSUA62VYFSGSPFOXKRIFNWMUGULPA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:50.034551511 (565h55m49.905634546s) [noise=true , address=77.90.31.10:4519 , piece_id=IY2ADV6FEPZS4DUJ4NUR4NBKXCCOLKBI5X6UMY6UKM53FNYP56EA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:50.034569975 (565h55m49.905653011s) [seg_pos={0 0}, upload=496177]: Segment reached optimal threshold of 13 pieces. Dec 29 05:24:50.034785406 (565h55m49.905868451s) [noise=true , address=77.90.31.16:4735 , piece_id=6QBDKLH4Q44DS7GHFT47KTPWKVBQDBPA2RJING4UGTJS6H5J4KEA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 29 05:24:50.034792780 (565h55m49.905875815s) [noise=true , address=77.90.31.16:4735 , piece_id=6QBDKLH4Q44DS7GHFT47KTPWKVBQDBPA2RJING4UGTJS6H5J4KEA, seg_pos={0 0}, upload=496177]: Long tail cancellation due to optimal threshold Dec 29 05:24:50.034914672 (565h55m49.905997717s) [noise=true , address=45.91.116.4:4939 , piece_id=YNKOKBCJSHNH2VCG3GS4XEAJK2W4VJG2RWLIZORPHHJMVVNASJGA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 05:24:50.034920803 (565h55m49.906003838s) [noise=true , address=45.91.116.4:4939 , piece_id=YNKOKBCJSHNH2VCG3GS4XEAJK2W4VJG2RWLIZORPHHJMVVNASJGA, seg_pos={0 0}, upload=496177]: Long tail cancellation due to optimal threshold Dec 29 05:24:50.035124131 (565h55m49.906207176s) [noise=true , address=45.91.116.7:4011 , piece_id=XQYPLTH3DKGGTJC34E4KAD2G2MHKZ4UZGREQDF5WOBRZXKE6MBCQ, seg_pos={0 0}, upload=496177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:49596->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:49596->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49596->45.91.116.7:4011: use of closed network connection Dec 29 05:24:50.035139350 (565h55m49.906222395s) [noise=true , address=45.91.116.7:4011 , piece_id=XQYPLTH3DKGGTJC34E4KAD2G2MHKZ4UZGREQDF5WOBRZXKE6MBCQ, seg_pos={0 0}, upload=496177]: Long tail cancellation due to optimal threshold Dec 29 05:24:50.035162253 (565h55m49.906245298s) [seg_pos={0 0}, upload=496177]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:24:50.035254890 (565h55m49.906337935s) [upload=496177]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:24:50.035260270 (565h55m49.906343305s) [upload=496177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:24:50.332008061 (565h55m50.203091126s) [upload=496177]: Done uploading segments... Dec 29 05:24:50.332032207 (565h55m50.203115252s) [upload=496177]: Upload finished. err: <nil> Duration: 5.643150216s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX..XXX........XX......X...X..XXX..X............X..X.X...XX.] duration: 25.057147747s trace: https://jaeger.dev.storj.io/trace/291592605b9f77be FAIL (2025-12-29 05:25:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 05:24:47.259947029 (565h55m47.131030064s) [upload=496177]: Starting upload Dec 29 05:24:47.260005089 (565h55m47.131088134s) [upload=496177]: Done starting upload Dec 29 05:24:47.260167499 (565h55m47.131250544s) [upload=496177]: Uploading segments... Dec 29 05:24:47.260245487 (565h55m47.131328522s) [upload=496177]: Got next segment. Inline: false Dec 29 05:24:47.260308928 (565h55m47.131391973s) [upload=496177]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:24:47.260316692 (565h55m47.131399728s) [upload=496177]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:24:47.506070944 (565h55m47.377153979s) [seg_pos={0 0}, upload=496177]: Begin upload segment... Dec 29 05:24:47.506119246 (565h55m47.377202281s) [seg_pos={0 0}, upload=496177]: Done begin upload segment. Dec 29 05:24:47.506160334 (565h55m47.377243379s) [noise=true , address=77.90.31.13:4627 , piece_id=MJ57NKNPOZBCRGV47V2O6MZ5AMUC7RUWOXHTE7YAW37OI6Q3UKNA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.506280624 (565h55m47.377363659s) [noise=true , address=145.79.227.45:4230 , piece_id=A4RFQL2LD42H2VNKH533LDSE2PEWYJNS6T3QZTDM6JL2Q4MWQ7EA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.506874013 (565h55m47.377957048s) [noise=true , address=145.79.227.6:4257 , piece_id=KH7JYCCPG7V7GFPSY7NX4K4GG7M2VV3JYVVOD7N2HI6YWDAHG4CA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.507123749 (565h55m47.378206794s) [noise=true , address=77.90.31.10:4519 , piece_id=IY2ADV6FEPZS4DUJ4NUR4NBKXCCOLKBI5X6UMY6UKM53FNYP56EA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.507524743 (565h55m47.378607788s) [noise=true , address=145.79.227.46:4603 , piece_id=2SSW7O2HHHV5KBGBNMAPFD2HCLZZDTMPTR3ENDLPVQD53GQ6VT3A, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.507902051 (565h55m47.378985096s) [noise=true , address=45.91.116.7:4011 , piece_id=XQYPLTH3DKGGTJC34E4KAD2G2MHKZ4UZGREQDF5WOBRZXKE6MBCQ, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.508284038 (565h55m47.379367083s) [noise=true , address=145.79.227.48:4303 , piece_id=H6JVKF76M4F24BRG73FSNKSESVZABZ27FA6ENLGPVSV4DBI3FHYQ, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.508670304 (565h55m47.379753339s) [noise=true , address=77.90.31.19:4842 , piece_id=KJ5EB6QDRPV7ZAEJMYH3SPG6WCL6HYZFYRQWKGRR5N55Q7UOD5LQ, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.509077389 (565h55m47.380160424s) [noise=true , address=145.79.227.15:4873 , piece_id=WJBG6SGY4GEYRBWPCC5AG6MGEUSOL2K7TB4V3JIQ5D6ZQ2EJ3YHA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.509520863 (565h55m47.380603908s) [noise=true , address=45.91.116.4:4939 , piece_id=YNKOKBCJSHNH2VCG3GS4XEAJK2W4VJG2RWLIZORPHHJMVVNASJGA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.509980498 (565h55m47.381063544s) [noise=true , address=145.79.227.42:4122 , piece_id=SLI6VYEI2DIFCDC4LVDHXTWPSBQ536RTZDVXQOTW6G6ZCJ2ENL3Q, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.510107250 (565h55m47.381190285s) [noise=true , address=77.90.31.16:4735 , piece_id=6QBDKLH4Q44DS7GHFT47KTPWKVBQDBPA2RJING4UGTJS6H5J4KEA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.510409185 (565h55m47.38149223s) [noise=true , address=145.79.227.41:4087 , piece_id=CN56ZHZNNIACHGPTPVTILJOIQBJRHMSNU34FVI5MNTGGELLN75AQ, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.510423222 (565h55m47.381506257s) [noise=true , address=45.91.116.14:4189 , piece_id=W64YUT6AQ5STTW5LX7VJDCGXSUA62VYFSGSPFOXKRIFNWMUGULPA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.510922733 (565h55m47.382005768s) [noise=true , address=145.79.227.21:4473 , piece_id=ZF4PYDJNAYOADT3CEANDQX6PMRKU3ZLKC6IPBZWOOOFMIAHFJ5BA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:47.511310150 (565h55m47.382393185s) [noise=true , address=145.79.227.26:4654 , piece_id=I7UEZZGDPNASYHWIBVNEXPBXAW55UE4JDNDK54XIU7I7OPZY2CRA, seg_pos={0 0}, upload=496177]: Uploading piece... Dec 29 05:24:49.197215253 (565h55m49.068298298s) [upload=496177]: Next returned nil segment Dec 29 05:24:49.197231054 (565h55m49.068314099s) [upload=496177]: Waiting for error group managing segments... Dec 29 05:24:49.478413174 (565h55m49.34949622s) [noise=true , address=145.79.227.45:4230 , piece_id=A4RFQL2LD42H2VNKH533LDSE2PEWYJNS6T3QZTDM6JL2Q4MWQ7EA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.498156930 (565h55m49.369239975s) [noise=true , address=145.79.227.26:4654 , piece_id=I7UEZZGDPNASYHWIBVNEXPBXAW55UE4JDNDK54XIU7I7OPZY2CRA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.499262935 (565h55m49.37034597s) [noise=true , address=145.79.227.46:4603 , piece_id=2SSW7O2HHHV5KBGBNMAPFD2HCLZZDTMPTR3ENDLPVQD53GQ6VT3A, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.501031092 (565h55m49.372114127s) [noise=true , address=145.79.227.41:4087 , piece_id=CN56ZHZNNIACHGPTPVTILJOIQBJRHMSNU34FVI5MNTGGELLN75AQ, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.517640107 (565h55m49.388723143s) [noise=true , address=145.79.227.48:4303 , piece_id=H6JVKF76M4F24BRG73FSNKSESVZABZ27FA6ENLGPVSV4DBI3FHYQ, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.518412017 (565h55m49.389495052s) [noise=true , address=145.79.227.15:4873 , piece_id=WJBG6SGY4GEYRBWPCC5AG6MGEUSOL2K7TB4V3JIQ5D6ZQ2EJ3YHA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.527292317 (565h55m49.398375362s) [noise=true , address=145.79.227.6:4257 , piece_id=KH7JYCCPG7V7GFPSY7NX4K4GG7M2VV3JYVVOD7N2HI6YWDAHG4CA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.538294827 (565h55m49.409377862s) [noise=true , address=145.79.227.42:4122 , piece_id=SLI6VYEI2DIFCDC4LVDHXTWPSBQ536RTZDVXQOTW6G6ZCJ2ENL3Q, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.642293897 (565h55m49.513376932s) [noise=true , address=145.79.227.21:4473 , piece_id=ZF4PYDJNAYOADT3CEANDQX6PMRKU3ZLKC6IPBZWOOOFMIAHFJ5BA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.674396034 (565h55m49.545479079s) [noise=true , address=77.90.31.19:4842 , piece_id=KJ5EB6QDRPV7ZAEJMYH3SPG6WCL6HYZFYRQWKGRR5N55Q7UOD5LQ, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.694408821 (565h55m49.565491856s) [noise=true , address=77.90.31.13:4627 , piece_id=MJ57NKNPOZBCRGV47V2O6MZ5AMUC7RUWOXHTE7YAW37OI6Q3UKNA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:49.947202917 (565h55m49.818285952s) [noise=true , address=45.91.116.14:4189 , piece_id=W64YUT6AQ5STTW5LX7VJDCGXSUA62VYFSGSPFOXKRIFNWMUGULPA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:50.034551511 (565h55m49.905634546s) [noise=true , address=77.90.31.10:4519 , piece_id=IY2ADV6FEPZS4DUJ4NUR4NBKXCCOLKBI5X6UMY6UKM53FNYP56EA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: <nil> Dec 29 05:24:50.034569975 (565h55m49.905653011s) [seg_pos={0 0}, upload=496177]: Segment reached optimal threshold of 13 pieces. Dec 29 05:24:50.034785406 (565h55m49.905868451s) [noise=true , address=77.90.31.16:4735 , piece_id=6QBDKLH4Q44DS7GHFT47KTPWKVBQDBPA2RJING4UGTJS6H5J4KEA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 29 05:24:50.034792780 (565h55m49.905875815s) [noise=true , address=77.90.31.16:4735 , piece_id=6QBDKLH4Q44DS7GHFT47KTPWKVBQDBPA2RJING4UGTJS6H5J4KEA, seg_pos={0 0}, upload=496177]: Long tail cancellation due to optimal threshold Dec 29 05:24:50.034914672 (565h55m49.905997717s) [noise=true , address=45.91.116.4:4939 , piece_id=YNKOKBCJSHNH2VCG3GS4XEAJK2W4VJG2RWLIZORPHHJMVVNASJGA, seg_pos={0 0}, upload=496177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 05:24:50.034920803 (565h55m49.906003838s) [noise=true , address=45.91.116.4:4939 , piece_id=YNKOKBCJSHNH2VCG3GS4XEAJK2W4VJG2RWLIZORPHHJMVVNASJGA, seg_pos={0 0}, upload=496177]: Long tail cancellation due to optimal threshold Dec 29 05:24:50.035124131 (565h55m49.906207176s) [noise=true , address=45.91.116.7:4011 , piece_id=XQYPLTH3DKGGTJC34E4KAD2G2MHKZ4UZGREQDF5WOBRZXKE6MBCQ, seg_pos={0 0}, upload=496177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:49596->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:49596->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49596->45.91.116.7:4011: use of closed network connection Dec 29 05:24:50.035139350 (565h55m49.906222395s) [noise=true , address=45.91.116.7:4011 , piece_id=XQYPLTH3DKGGTJC34E4KAD2G2MHKZ4UZGREQDF5WOBRZXKE6MBCQ, seg_pos={0 0}, upload=496177]: Long tail cancellation due to optimal threshold Dec 29 05:24:50.035162253 (565h55m49.906245298s) [seg_pos={0 0}, upload=496177]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:24:50.035254890 (565h55m49.906337935s) [upload=496177]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:24:50.035260270 (565h55m49.906343305s) [upload=496177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:24:50.332008061 (565h55m50.203091126s) [upload=496177]: Done uploading segments... Dec 29 05:24:50.332032207 (565h55m50.203115252s) [upload=496177]: Upload finished. err: <nil> Duration: 25.057175329s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X.....X.....X........X..X...X....X....X.........X...X.......] duration: 4.0099684s trace: https://jaeger.dev.storj.io/trace/159c3b7a9a64e4e4 FAIL (2025-12-29 05:25:30 UTC) orbiter warning: context deadline exceeded Duration: 4.009995011s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)